I know what you mean. Just looking at me, I appear pretty normal. But I have both mental and physical ailments that put me on disability. It is frustrating. No one has ever said anything to my face about it, though.
I am reminded of a story a friend of mine tells. She's on disability, but nobody could tell by just looking. She has a placard for disabled parking. One time she parked in one of those spots, and some lady came up to her and told her off about it. My friend calmly explained to her why she was disabled. The lady huffed, "Well, you ought to wear a sign that says that!"
My friend's teenage daughter was there and heard the interaction. She responded to the lady, "In that case, you need to wear a sign that says _ITCH on it!" I thought that was great!