"I have heard them say stuff about me a couple of times"

This is really rude from their part - not to mention how it is not professional *at all*. I would take it up with your T if you do not feel up to confronting them. It is none of their business whether you want to be in therapy or not and/or how often you go.

If you can afford to go to T, if you feel you are getting something out of therapy, then by all means keep going. But my advice would be not to let them get away with bad-mouthing clients. This is simply not acceptable.