Oh, I know what you mean! I have cancelled, or not showed up, to doctor appointments for the same reason. Sometimes it was because of my depression.
It's also been difficult for me to do stuff with my family because of my anxiety; my family always took it as me just not wanting to be around them. My sister in law has difficulty doing a lot of family get together type stuff, too--but I'm not sure if anxiety plays inot it or not; she does have some physical issues going --but you wouldn't know just by looking at her; so a lot of times, she caved in to my family's pressure to hang out. But she finally had to put her foot down (kindly, tho) and they backed off. They started backing off on me around that same time, too.