The end result is that I'm hugely triggered that he just left. What if I say something in the future he deems as not perfect or ok and he leaves then too?

He just left and I don't feel safe anymore.