Hello, Don'tStealMyBacon (digging the username by the way).
Honestly, being that empathetic is both a curse and a blessing: a blessing when you're among friends and wanting to know their pain, and a curse when you're among stressful or difficult people.
Consider learning some coping mechanisms for this level of anxiety.
This article may help.
I also have a "gift" of picking up secondhand emotions, and the way that I cope is by taking a second to check in with myself. Why am I stressed? Is it me, or this other person? If it's the other person, then you can start to look at them with some measure of concern or sympathy, and not resentment. In my experience, taking responsibility for my part in my discomfort works wonders. And then I can laugh it off.
I hope this helps, and I hope you stop feeling stressed
Good luck!