I wouldn't say there's necessarily anything wrong with you just because you're not attached to your therapist.

You could however still have attachment issues that need to be worked through. I personally find it very hard to attach to anybody, trust anybody, or let anybody into my physical or emotional space. This is due to abuse suffered at a very young age and to me that is a problem. Only you and your therapist can decide if you have similar issues that you want to address.