I can empathize, I'm turning 24 in a couple days and I'm in a similar situation. Minus the thing with your mom I've had pretty similar experiences. My first, I guess you could say proto-girlfriend in middle school was an absolutely horrible human being, incredibly verbally abusive and controlling. I have aspergers too and can't tell when someone is flirting with me unless it is incredibly obvious. and when it is obvious I get this overwhelming anxiety that has made attempting to find a relationship extremely difficult.

I asked my university's therapist I've been going to for anxiety about it and she flat out told me she had no clue how to help, so I'm just kind of stuck and I don't know what to do about it anymore. Best of luck, hopefully both of us can get past this issue.