I had to think long and hard on this one, mainly I agree with Rose on this, except for one thing.
If he gets earache every time he speaks to you he will and has moved further and further from you.

Anyway it's a moot point now, I would strongly advise just accepting this isn't what you think it is, and work on letting go.

I hope you have the strength and support to do what's best for you and your young one. Take care.