I agree with Pfrog, if you consistently start declining their invitations, eventually they'll stop asking.

If you're planning to return to work in the future, I wouldn't personally tell them outright that you permanently don't want to hang out with them. That could add to anxiety on returning.

As others have said, sometimes white lies are better to make everyone feel less anxious. It's not like you're telling outright lies to try to manipulate or exploit people, they're just useful tools in the workplace to try and keep people we don't want to be close to at a distance without hurting their feelings.