I agree that his behavior sounds inconsiderate and disrespectful. He should be aware of your schedule and coordinate things accordingly.

I don't know his personality is like. Some bosses are not open to any sort of discussion and will not take anything they perceive as criticism well. I guess only you know what your relationship is with him, and if trying to discuss it with him will get results for you.