I'd ask him who else he has taken to lunch to have an official bonding experience. Then when he can't come up with an answer, I'd say that you like your job, like him as a boss, and don't want to mess things up by getting overly familiar- it would seem inappropriate. Then if he keeps pressing you, get more firm, tell him you are not interested in him. If he still doesn't back off...IDK, you'll probably end up having to get another job.