I keep thinking about this, and I'm really wondering if part of it isn't about safety. Some of us were badly injured (emotionally and/or physically) by others violating our boundaries. So, boundaries being respected represents safety. When he violates that time boundary, maybe it triggers a fear that other boundaries get stepped over also?