It isn't your fault at all that you struggle trying to have 2 very different relationships with this man. What he's done to the therapy relationship is overburdened you.

Your story is like a testament for why a dual relationship doesn't work.
You are getting therapy. To do that, you need a therapist who is just a therapist. I hope you will decide to find such a therapist to work with.