I think you said exactly the right thing to the rude guy. The only suggestion I would make for next time, if a next time happens, is this:

"I wish you were right, but unfortunately half the time..." and the rest from your sentence.

This is less confrontational, because there are no "no" or other forms of explicit negation ("isn't"), but instead the negation is implicit and subtler ("unfortunately", "I wish you were right").

Also, "I wish you were right", while being non-confrontational, is subtly condescending, so it would actually get your point across better than the sentence you used last time.