You have done a lot to make a better life for yourself in the last couple years, splitimage. Please do not let others threaten what you been able to accomplish. Letting others define you is so counterproductive.

As reg12 suggests, maybe working with a different group would be better. Good luck.