Sounds like he is playing games with your mind. I suggest reading: THe Verbally Abusive Relationship by Patricia Evans (not saying he is verbally abusive), but this book should be read by everyone on the planet, in my opinion......1 in 3 women are living with abuse.

You need to do what you need to to protect yourself....what do YOU want?