God yes... working in a job like that could be stressful for anyone. Some thrive on the hectic pace but for someone with anxiety and depression...x50. Working is always preferable. There's something about working that makes you feel better about yourself.
In your case tho I'd listen to the doc. If you qualify for SSI then great. That will give you a break and you can have time to look into other lines of work. There may even be job training or education opportunities that this will open up for you. When you're ready for work again maybe you can get a place of your own...assuming that you live with your mom. Good luck