Yes, that sounds frustrating. But you haven't done anything wrong in the situation, you're consistently doing the same thing, which people can plan for in advance! You're being responsible, he's not.

RJ