Are they formally divorced now? You say he moved away but it does not sound like he moved very distant if he is still trying to arrange to meet up for lunch, etc.? That something is difficult for us does not mean we are to allow our difficulties to spill over onto someone else as he is doing with his previous relationship onto you and this relationship. He does not sound able to commit to you (yet) at this time and I would kick him out until he is interested in doing that as a grown man.