I was diagnosed with Asperger's, plus my own mother committed absolutely heinous acts of abuse against me (not just because of my Asperger's, but that was one of the reasons). I empathize completely with you. No one deserves to grow up in such horrendous circumstances, and for aspies it's even more vital that they get the basic understanding and compassion that others take for granted. Instead of comforting me when I cried, my mother laughed at my suffering. When all I wanted was a hug from her, she slapped me across the face and beat me senseless. When I cried out for help, she called the pigs on me and had me arrested for things I didn't do.

Pardon me for making it about myself. I don't know the exact circumstances of your situation, but whatever they were, you did nothing to deserve any of it. Growing up in this kind of household and having it be all you know can seriously harm your mind. If you can get the metaphor, it seems as if you are let out of the starting gate with an debilitating injury, yet expected to preform just as well as someone who's fine. You have every right to be easily frustrated like you said.

I strongly suggest that you discuss your feelings with your therapist, and also discuss what your options are for healing from the past. Ask about any social programs and groups or whatever resources are available to you that you think you would benefit from. Also, I would be remiss if I didn't tell you to pay attention to your therapist if you feel that (s)he is patronizing you - invalidating your feelings or treating you like an some kind of incompetent. The very definition of their role is to help you, and if they don't listen to your voice then they are contradicting their own job description. I say that because in the past that happened to me, having to be a patient of a manipulative psychiatrist. You owe it to yourself to be in command of your own care so that doesn't happen. You deserve to be listened to.

In the mean time, take care of yourself. Lay low and do little things that are beneficial to you. Self-care is an underrated thing in life and important regardless who you are. I know that most aspies have very specific and passionate interests, so I recommend that you involve yourself in activities and research pertaining to whatever interest you might have to help make yourself feel comfortable. You deserve a positive environment that you can feel confident in, so that you're able to make efforts toward realizing whatever your ideals are for your life.

I hope that you can make sense of my long and rambling post, and can extract at least some information out of it that you might find valuable.