We always have to remember not to take things personally at work no matter how much you deal with difficult people. It'll be hard, but since this guy isn't your boss, and you already told him no, bring it to the attention to your boss or someone near that level who works in your team. It should at least alleviate the stress and frustration you are harboring due to this guy. Stay strong, you'll be fine.