Being "surprised," would, to me, make it sound like a set-up, Like you knew all along that it was the wrong time and were testing him?

In that case, I would have presented myself first. "I really got the time wrong last week!" and then either tease him, "what's your excuse for why we started early?" (which also links you together and shares the experience as if it was a happening on both of you all's parts) or continue on about myself, "I didn't realize it until after I left" and then just "wonder" "Did you realize it was half an hour earlier than our scheduled time?" That would keep him "humble" :-) without accusing him of anything "worse" than you're accusing yourself of.