I think tying it to your past is part of it but honestly I think as often as that is happening it would make almost anyone anxious to a degree. That's just not a good environment to be around everyday, let alone be woken up by it regularly. Are you in a bad part of town? I couldn't say, only you know but I can say that the apartment complex needs some security or policing.

I'm sorry for your situation, I hope it gets better or you get a chance to move to a better place.

*hugs*