It can be awkward, but it sounds like you will need to tell him you aren't interested in dating, I think Divine's advice is spot on - you can also wish him well for the future, but keep it short.

IMO you did not act badly, you were caught off balance and trying to act in the least upsetting way to someone who was being pretty full-on for a first date.