Yes, I can completely understand this.
I always viewed my hallucinations and 'fantasies' as a big part of my life. As part of my personality so to speak? I can't explain well, sorry.
So when people just want to get rid of it and call it fantasy...it is annoying. You think that they are ignoring a big part of your personality?