Your partner's needs are not being met. Him asking for affection is not an unrealistic expectation in this relationship; given that you haven't painted him out to be some kind of monster.

Relationships are give and take. And sometimes we need to set out of our comfort zones. If this is truly something he wants, can you not meet him halfway, not for you, but for him.