I think that statement was meant matter-of-factly, but you may be hearing it as an angry statement. What do you think?

Accommodating and tracking scheduling exceptions is extra work for the boss, and I can empathize with that, too.

Sometimes we are limited by work schedules. Frustrating, for sure.