I think I would just leave it - I can sympathise with feeling guilty when you stand up for yourself, but I also think you did well standing up for yourself this time. It doesn’t sound like he respected you as much as you deserved, if he was being vague and ignoring your desire for more straightforward communication - that’s never a good sign, neither is feeling unsafe when you’re talking to them.
I don’t have much advice for if you see them in public, however. Maybe if you do see him, go with what your gut says? Sorry I don’t have great advice either! Best of luck, anyway. But in my opinion it would be ok to leave this as is.