In my view you could just explain the truth of the situation, which is that you don't want to get involved in the organisation of this group. If you feel that you also don't want to join their social events, then again I think it would be simplest just to tell the truth, i.e. that it isn't quite what you are looking for. I don't think that should offend anyone. You really have nothing to feel guilty about