Your first obligation is to yourself. You are not responsible for the feelings of another person. If you must go, then start planning now. Make it your new project. If you love your job, perhaps you can live near the job, or else move back to your original home town. Or even go someplace completely new, that you've always wanted to see.

Sometimes it's better to just leave a note, and then go. Sometimes, it's the only way.

Life goes on, he will survive without you, and you will not survive if you stay there. So you must go.