If someone said that to me, I would first be embarrassed and then I would be upset that they would be so intrusive to say that to me.

You are viewing it as being caring, but just reading it I view it as putting a big spotlight on the fact that my family purposely excluded me from a gathering.

Next time, just leave it as "how are you doing?"