I think ideally, the phrase is used when a person doesn't understand the difference between their own reality and (what I call) "consensual reality" or "group reality" - you can't tell the difference between what only you see/hear/feel and what the objective observer would.

But yeah, I always felt like it was pejorative when it was applied to me. I think often mental health professionals go beyond the use I said in the first paragraph and apply it to people who see the world in a different way. It's unfortunate.