I wouldn't worry too much about it, to be honest. Some people like to take it slower than others. Did the date go fine otherwise?

Perhaps after your third date, if he doesn't make a move, you could go in for a kiss? Just a thought.