Good for you for being assertive, though I can understand your emotional dilemma. I’ve been on both sides of this fence, I’m kind of ashamed to say… I learned from it, though, and an apology from the other person (who was in your position) probably wouldn’t have improved the situation at the time. Sometimes consequences are the best teacher.
It’s understandable that you’d worry about what they might have done afterwards, but like others said, you can’t control their reactions. You can only be mindful of other’s feelings so much before you find you’re putting your needs below everyone else’s.
I think each time you enforce boundaries it will feel a bit better, too.