A lot of times you can form sort of a self fulfilling prophecy. You worry about the relationship, you pull back just a bit in self defense, and he senses that something is wrong but he isn't sure what so he pulls back, you sense him pulling away so you pull back further..

I'm not sure what you should do about this though... seems he is still hung up on his ex for some reason.. could be anger, or he could have feeling for her still, or something silly like she has his favorite sweatshirt and he wants it back..