I'm sorry to hear this. It sounds tricky.
I wouldn't worry about coming from two different places in life.
Just a few questions: Have you voiced your thoughts to him much? I understand you've asked him at least once. Did he not give a straight answer? Does he communicate much with you, such as his or your feelings?
I think some people are genuinely like this, but it's impossible to know your whole situation. Do you do most of the "work" in the relationship?