I can only imagine how it must feel to be photographed without your permission. I take camera in hand often to catch daily life, but I would never take a photo of someone without asking permission first.
Some folks are out and out obnoxious it's true, but some folks, really do care and try to educate others about the homeless situation. Maybe if more are educated, there will be less ignorance
I wish you well....and much safety!