Thank you to everyone for your answers.
Here's what I meant by being angry with myself for not letting it go:

In the encounter, I could have simply let it go, meaning not responded in any way whatsoever. but I didn't, and that is what bothers me the most.