It's good that he called & you had a chance to talk. If it's not a major hardship for you, I think you will probably be happier in the long run if you go. Now ... how did that strike you?

You alone know whether you should go. What does your gut say? You could discuss it with him, too.