I'm very particular about physical contact and yes, I've felt what you're describing on more than one occasion.

For me, it can be one of three things causing the reaction:
1 - I feel trapped. That's the anxiety talking, I know. But, I feel like the other person is just too close, they're intruding in my space, and if I make physical contact, I won't be able to get away.

2 - Sometimes, I'm just being very sensitive to my environment and the other person's presence feels almost like an interference pattern with my own presence. I feel almost repelled by them (like you described - trying to shove two like poles of a magnet together).

3 - I need to feel mentally "connected" to a person before physical touch is comfortable for me. If someone I'm not feeling connected with on some other level, right at that moment, touches me, it feels very intrusive and unpleasant.