If you are planning on going back to the job after your leave, it might be better to not tell them you never want to hang out. I have always declined those sort of workplace group invites and only recently realized that they are actually an unspoken requirement.

You can decline 90% of them with excuses about 'I have too much going on right now, maybe I can meet up in June' or whatever, but I would not tell them that you don't want to hang out ever. I am also very non-confrontational, so take that with a large grain of salt