If I were you, his actions would not sit well with me. The way he told you, though, it seems like he thinks it wasn't too big of a deal. I think you should express to him how his actions made you feel.