I understand where you're coming from, I think. I was the same way... good news is that I'm gradually finding it easier to have belief in myself and to talk to people without assuming they're thinking the worst of me or are going to leave (which has happened to me a lot). So at some point it can happen for you too.
It sounds like a self esteem/confidence issue. It's hard and time consuming, but it's possible.
I've started a confidence building group a while ago and I believe it's helping. Do you ever talk to your T about this? Maybe self esteem is something you could bring up?
Take care