It comes from my heart. It feels like a loving thing to do.
On the other hand, he's not very reassuring with me, so it makes it seems like he has no feelings of affection for me.

It just seems like a natural thing to do when you care, though I realize people have different ways of showing care.