That is out of line. Not only did he wander off in your marriage but instead of trying to mend the relationship, seems like he's only worried about the sex. I don't know you nor your husband so I cannot go into great detail with my response but from what I am reading, he is not doing a great job at trying to fix anything. Emotional abuse can be possibly worse as it can affect your mind - which can affect your life. He doesn't seem to respect you at all. Please do what you truly feel is right for YOU.