You have the right to draw any boundaries you like and your friend needs to respect them.

If possible, keep reminding yourself of this fact, and don't worry about whether he will have hurt feelings or not.

Sometimes fear is much worse before we do something, then once we actually do it, things aren't as bad as we expected.

You count, you have rights, you are important. If you don't want to be touched, and he keeps touching you, no matter what reasons or excuses he gives, he is not respecting you.

I hope you keep listening to your own feelings and experience, work on this in therapy, and tell him clearly what you want and what you do not want.