There's nothing wrong with avoiding groups of people; especially if the groups of people that you're trying to avoid are becoming a threat to your well being or happiness.

Many people that succeed in life will tell you to associate with other people who are like you want to be and avoid people who have the potential to drag you down.

For example, if you are trying to succeed in running a business, surround yourself with top performers that want to succeed and avoid people who complain about being poor but are poor due to their own choices; if you are trying to be an athlete or get in shape, surround yourself with other athletes who will push you to do better and avoid people who would become a negative influence on your fitness goals; if you are trying to improve your mental health, surround yourself with people who are seeking to improve themselves any way they can and are trying to be positive and avoid those who play the victims or pity parties, drink, do drugs or whatever else.

See my point?

So no, there is nothing wrong with being this way. In fact, you are doing things the right way.