Never sacrifice your well being for socializing. Doing what you need to for your own mental and physical health is not selfish.

what it selfish is a so called friend expecting you to sacrifice your well being so that you can socialize on their timeline.