I experience the same uncomfortable feelings with hugs & intimacy. I even feel uncomfortable receiving hugs from family.

I also become a little more relaxed when I'm drunk, but I still get a very uncomfortable with intimacy.

It doesn't really bother me though, as I don't desire this kind of physical interaction. I just do it a few times a year to appear normal.

I wouldn't worry too much though, as you said, you don't mind being affectionate with people you are close to and comfortable with. Surely that is all that matters? I understand though you probably don't want to come across as rude by telling someone to stop hugging you or whatever else. If a girl asks me for a hug, I generally just do it, in such a lacklustre way its even more awkward than telling them not to hug me. I'm lucky though, in the sense that even though there are girls that are attracted to me, they are intimidated of approaching me as I come across as cold and keep myself to myself.