I'm sorry about your son. That sounds very challenging, and could potentially contribute to anxiety reactions for most anyone.

Do you have a caregiver support group by chance? People who can relate directly to your situation, and offer meaningful support? Even if you aren't the main person who's caring for him day to day, you're potentially the person he sees as closest to him- in that, he'll likely see you as the person he's most comfortable with, and may direct more of his unpleasant behaviors at you. Caregiver burnout is a very real problem for family members, even at a bit of a distance.

It sounds like you have some good boundaries with him already. It's okay to have those, and it's also okay if you look after your own needs first too. His issues may be beyond his control, but you're treating him as well as you can without compromising yourself.

Big ((((hugs)))) for you. I hope you are having a good day.