So, since you don't do the obvious things that annoy people, I think what you may be experiencing is… you're interpreting what people are doing as dislike--specifically--for you (even when it most likely isn't) and you've likely run into some aggressive people who treat you the way they treat most people (but I am guessing they avoid the ones who would give them a poke in the snoot for their nasty behavior).

How do you feel about yourself overall--not in relation to others--but internally, just about yourself? One thing I do to help me get through the day at a very stressful work environment is as I am walking towards the building I think of something wonderful and I end up walking into the building with a smile on my face. Would you try coming up with 5 positive things you like about yourself, and saying them every morning? And also try purposefully thinking about something you find joyful as you approach what might otherwise be a place where you have encountered these people?

P.S. You're not a terrible person. Truly terrible people don't care if others think they are terrible.