I wouldn't apologize to him, unless you wanted to discuss this further with him, and then maybe do a sort of non-apology of "I'm sorry if you were hurt by what I said, but I thought it was important to bring up the fact that I seem to feel better when I don't see you as often."

Honestly if that is true (that you feel better when you don't see them) then that might be a signal that something needs to change in therapy.