Perhaps the people around you wouldn't be as hurt as you imagine?

Could you see that others might think you're prioritizing rather than setting out to hurt? There's a difference...
Could you look around to see if there's a place that's a better fit for you in your business? Even just testing the waters might be a good thing to do...