I think that the first and most important thing to do is apologize to X for ditching him.

Next, you could, in a kind manner, explain whatever boundaries you wish to have with regard to the frequency and nature of your conversations.