In my opinion, you are not wrong in thinking that.

However, he is going to have to say something if you choose not attend the family events.

You want to maintain confidentiality. He wants to be honest with his mother/family, and not lie to them.

As I see it, both sides have reasonable desires.

What should he say to his family that is honest about why you are not coming to the party?