Why don't you try a self-imposed period of no contact? If he texts you, calls you, contacts you in any way, ignore him for...I don't know...a week? Two weeks? It's up to you really. Force him to see that you are serious.

I've been in a relationship before where we were both pretty clear that it was not to be that kind of relationship. No feelings involved. But they happened and it threw us both off. It sounds as though he might have even been the one that let feelings come into whatever it is that you can define your interactions as; introducing you to family, friends, etc.

You can stay in this situation if you feel it is mutually beneficial - you did say that you needed a distraction after your divorce. However, from what I can tell, your needs now reach beyond a simple distraction and this man will not be able to provide you with what you are looking for. Don't contact him. Ignore him. You don't owe him anything further - including an explanation for why you're ending things.

Strong feelings are tough to overcome, but they will be worse if you keep going back. Take it from one who knows...