That really is just rude. If he starts like this then what would he do after some time. I'd find an excuse to cancel and not have anything to do with him.
Take yourself out to a nice dinner or invite a friend to go with you. It's really good that you are able to see how wonky this man is before you get involved. Good for you!!