I feel the same way at least to some degree. For me it was a trust issue. A part of me always trusted their motives, worried about hugging too tight or lightly, of ending them with my perfume, etc. With men I always feared there was some sexual component. I LOVED hugs from women I trusted but still had my fears.

It changed when I met T she hugged me first weekly for so long I was able to put my fears aside. Then I joined a group of women who always hugged. So gradually I have become more comfortable with hugs. It will likely always be in the back of mind.