I teach a bully program and what I have found is that he is bothering you because he is getting a reaction from you. You are reacting because you feel bad about yourself and he is triggering this. The kids I work with get the bullying to stop by not reacting anymore. They stop reacting by realizing that they are not a bad person. When they truly feel better about themselves, the bullies can't get a reaction out of them and they stop because they want the reaction and if they can't get one they aren't going to waste their time. So if you can work on why you feel bad about yourself and why you should feel good about yourself, this will help a lot. I can help you here if you want.