I think when we walk away we think about how scared we might be because someone is acting irrationally and we don't know what to do and that can make us feel ashamed, like we "should" fight/"stand up for" ourselves.

The guy was a jerk and, you said, a drunken one! His comments were nonsense if you think about it; you aren't planning to go to jail anytime soon are you? He may be going there but you didn't have plans for that did you?