Well that can be for a lot of reasons and I don't know if you'd be comfy talking about that stuff. But it's important to start working on this sort of thing because misunderstandings like this happen all the time and a lot of potentially toxic situations can be avoided like this. Note that the guy by just walking off has created a murky situation already. And he didn't have to say a word!

Personally I always saw this as a necessary part of life as people will cross your boundaries intentionally or otherwise. Speaking about it lets people know what you're ok with. Don't think of it as confrontation. It's just a necessary way for everyone to get on the same page with you. It's hard probably because you aren't used to it. All you can do is just speak from the heart. Don't try and prepare some response because you'll get REALLY nervous... Just shoot straight and be kind about it.

Maybe they won't like it but you have to remember that this is as much about you as it is about them.