Talking to the therapist about that comment and how it affected you actually seems like really good practice for the whole work situation! Just because someone doesn't have the perfect response every time doesn't make them a bad person or bad at what they do. There are more options than all or nothing, stay or go.