Your therapist should not have said that to you. Its quite possible that you are right which would explain a lot of what you said such as not wanting physical contact and not wanting to get close to you. It might be worth changing therspists. Its wrong of your therspist to doubt you, you need to be able to have trust.