I am a big mess, if you look at my labels. I am Avoidant as well as BPD, but the BPD is a much stronger trait in my personality (to me, anyway). But, yes, I have a big problem with both accepting and expressing affection towards others. I feel extremely uncomfortable with both.
My family was extremely negative and non-affectionate, maybe that's my reason. My POS brother is the same way in that respect, but he's an abuser. My sister is very affectionate with her family, but she's AvPD. Don't know if that means anything, but that's how my family ended up.