It does sound like this is way more drama than is warranted. It should have been enough for you to say, "Gee, Honey, I hesitated to disturb you. I wish
I had known you would like to have joined us." The END.

No future checking with him every time you do something. No self-reproach. No big fight. You didn't know. There's no way you could have known. That's it. Over. All 3 of you can do breakfast another time. If he stops mentioning this, as he should, then you stop too. Both of you need to let go of this. It's no big deal. Major trouble will certainly come your way in life, as it does to us all. Save your angst for when it is warranted.