It sounds to me like he enjoys pushing your buttons. Make your decision based upon your own needs and goals. As John said there will always be someone you find annoying in the workplace. Who knows who you'll end up with if you decide to leave because of him. Could be someone much worse.