From personal experience, I have accepted invitations to do things socially, then I backed out of them because I was anxious, and then ended up totally regretting that I didn't go through with it.
I sympathize greatly with your feelings of anxiety, but I think you'll regret it if you back out of this weekend. Think about it - this is just one weekend. You'll have plenty more weekends in your life. This is a great chance to spend time with your mother, and who knows? Maybe you'll have a lot of fun.
As sugahorse said, perhaps your pdoc can give you some meds to cope if you find yourself getting too anxious to handle. I hope you have a great weekend.