Here are some ideas for replies:

—You are right, I do need some time alone.

—You are free to get time alone as well.

—If you don’t need time alone, so be it. I do.

—If you choose to do nothing during your time without me that is your choice. That is up to you.

—I am not going to tell you in any detail about my conversations with friends.

Keep repeating your statements for as long as needed.

In other words, what if you are not defensive, just civil and unapologetic with him?