Your mental well-being is so much more important than a bit of extra money. I think you're making the right decision, the environment sounds toxic. Focus on school for now and if you want to get another part-time job in the future, go for it! Hopefully it will be better than this one.