I am no shrink but this relationship does not sound very healthy to me. It sounds sort of like he may be using you to fill in when he has no other girlfriend. I'm sorry if that isn't what you wanted to hear but maybe it's for the best. You need to just end the relationship for good and all and find a man who will love you in every way and not just sexually.