Have you spoken to him about how you feel? Maybe he actually isn't noticing the damage his distance is causing.
And as others have mentioned, I would start planning. Looking into what options you have available.

Goodluck...