Glad you called him on it, gave him some very direct feedback. It may improve the relationship.....! But, you may feel a bit worn out from having to train him?

I often feel my T is behind a fence or wall, to keep himself safe from my "fondness" for him. But his humor does help connect us, really helps.

My T is super polite, helps me with my coat, holds the door and walks me out to the main lobby. 180 degrees from your guy. However, it's a group office and I think they want to make sure clients get out of the back hall that connects the offices.

BUT on the original idea from SanitySeeker - countertransference
In the initial year of work, I often would touch my T on the arm while talking. He never said anything that it was OK or not OK. In January (?) I finally asked, since he seemed a bit standoffish. He said "I believe it is OK for a person to touch someone on the arm"....??? Then said It's OK for you to touch me on the arm.
The following session I reminded him of his lame initial response. Then he admits he really IS uncomfortable with it!!
I asked him Why did you permit it? How does this teach me about boundaries? He said, it seemed like the right thing to do at the time.

3 months pass - In my mind, I developed the theory that he permitted it because he LIKED it, and was uncomfortable because he enjoyed it. So I asked him again WHY WHY WHY

Finally he apologized, admitted he wasn't taking care of himself by letting it go on. He felt bad that he was already putting so many limits on me, and did not want to set one more. He altered his behavior, permitted a personal boundary violation, to avoid seeing my sad or angry or hurt reaction. Classic countertransference! But not erotic. So glad I asked again vs. continuing with my internal theory.
He says he knows better for next time. He says this is why it's called a therapy "practice"...

So, now I can annoy him by putting my hand very close to his arm and saying "I'm not touching you" .. .. ...

But maybe revisit the "you let me go over because you love me" comment, he may have retreated to a safe head space.