Just a thought if both of you pull back because each of you think the other doesn't want affection, etc. - I sense that the gap between what you want and what you get may grow bigger and bigger.

I would say if you feel it do it..