Sorry you're having problems!

Setting boundaries has always been difficult for me, too. But I've got better. Let this experience go, if you can, and maybe use the next time this raises its head as a chance to practice some simple ways of saying no.

You could have a couple always ready. They don't have to be confrontational.

I can't (and move away).

Somewhere to be (pointing at a random place in the distance, and then strolling towards it).

Not right now (appearing deep in thought. You can stay still with this one!).

I'll get back to you (and don't).

They are just a few examples. Maybe you could add more, and practice saying them at first with people you're fairly comfortable with.