Personally, I would want more out of a relationship. It sounds like he's more interested in pursuing someone else and keep you as a side distraction to me.