Well it is rightfully their money that they've probably earned, so it is immoral. If you, however, ask them for it, because you are genuinely desperate for some, and they refuse, then karma would strike the wealthy person for not giving to someone politely asking in need, even if it wasn't even a mackerel of money.
But often I think that karma works whenever it irritates me, so like even though it may not because people annoy me not deliberately, secretly I still think karma works against them. Another way of thinking about it is if you're prepared to do something bad, karma might affect you regardless for having that kind of personality. So if you're a thief, that shows you're not prepared to work for your income, or that you've given up using charities that are there for you, making you an easily relinquishing person which is karma in itself because it's a bad personality trait.