I went to uni in the city, and the noise, smells, street harassment, etc, really made a negative impact on my mental health/overall quality of life. I felt like I had to put on a mental "armor" every day, even heavier than the one I normally wear. Not going downtown daily anymore, really centered me, and helped me get into a more manageable place. I don't mind visiting occasionally, now.

When I felt overwhelmed, I'd try to find somewhere quiet downtown, like a public library, or garden.