Your situation is very tough. Having this guy right nextdoor is making it hard to recover from being hurt. As a caregiver, you probably have limited opportunity to be out among others and meet people. I was a caregiver for a number of years. I know how much that takes over your time and energy.
Your mom is lucky to have you. Her dependency may be a bit unfair to you. Perhaps your mom may be eligible for some hours of caregiving that she might not have to pay for. It takes a lot of investigating to get answers to that. Your local Department of Aging might be able to give you 10 to 20 hours of a home attendant coming by. Medicaid also has a program, but it tends to involve going on a waiting list. If your mother has the means, you could consider hiring someone to give you a break. What you're doing is really too much for one person. You need a bit of freedom to be able to expand your social life. Caregiving does get quite lonely. But I commend you for the wonderful gift you've given your mom. I hope she knows how lucky she is.