Are you gaining anything positive from your interactions with these people? If not then I’d disengage. Getting fit is something you do for yourself not to spite others. If you get drawn into this it’s likely going to result in more negativity being drawn into your life and not good for your mental health.