Next time you see him, just ask if he meant to send of note or not on the scheduled day. And then wish him a happy belated birthday. I am sure he will be reasonable and even say how he actually feels on the subject. He knows he needs to earn your full trust, and that is something he is trained to do as a therapist.

Just breathe for now. And wait until the next appointment.