You avoided him not because he is in a wheel chair but, because he is a lonely guy who begs for money and you weren't in the mood to deal with it. You're entitled to your space.

Nothing wrong with that nor do I see avoiding negative situations (judged as such based on past experiences with certain people) as contributing to stigma.

Letting people invade your comfort simply because they might be mentally ill or disabled is just as bad as avoiding people for the same reason.