Your boss is no different than 90% of the so-called authority figures out there. They only care about themselves.

If he expects you to stay after 5 to wait on him than he should pay you for your time. Otherwise, he's being selfish and you shouldn't have to put up with it.

So no you're not in the wrong. He is.