I'm sorry you're having this experience with people. While I don't have a good response (I hope others have excellent ideas), I really encourage you not to say that's rude even though it clearly is. If you're trying to build bridges, that might not be the thing to say. Some kind of deflection or wry humor might be better. JMHO.