I agree, you don't need to disclose all of the reasons why you don't want to go. If you're unable to express how you feel would your mother respect the fact that you need "time out" from your brother? It does not sound like you'll be ready any time soon to see him.

No, it's not that you don't want to be a part of the family. And if your mother jumps to that conclusion I'd say she's being short sited.

If you can't think of a valid reason why you can't go (work etc because she'll think of a way around it) - I think that, as an adult, it is okay to state that you do not wish to go for the time being.