I would be upset to.

What I would do is make a phone call to the ex. I wouldn't even tell your son about it. Your ex might be doing this to get back at you. Tell him if it happens again that you will do supervise visitation with him. Safety for your child comes first, so if that means making someone upset then that is just what you have got to do.