You should try talking to him about it. Such conversations might be difficult, but there's no other way to stop it. If you wait until one of those impulse times and are like 'no stop' he could take it a personal insult, so you should talk about it in a non-sexual time and explain to him how you feel that you like him, etc., but just want to go slow.