I have no doubt that you are suffering. I have no doubt that every human being makes mistakes. I even accept that your husband might not be an angel and yes, you do have the right for love and happiness.

What I do not accept is cheating. I believe that if someone is in a relationship/marriage that is supposed to be monogamous but it's not working, said person should fix it or finish it before seeking love/sex/ sharing your dreams or whatever it is you do elsewhere.

I don't know your full story but you are asking help with your own confusion and emotional issues while you are most likely be causing the same amount of pain and confusion to your husband, knowingly.

I find this attitude extremely selfish.

My best advice to you is sorting out or ending your marriage first. I understand that you are asking for help regarding your lover but in my honest opinion, many of the issues probably stem from the fact that you are a married woman and not necessarily from any mental health issue your lover might have. The fact that he withdraws after each date could well mean that he might just have what is called conscience.