"[You] don't want to leave." Surely that implies something to do with your feelings?
What is it that makes you want to stay with him?
If you haven't done so already, tell him that you don't want to leave, but that you want to make sure you're staying with him for the right reasons. It might help him to know that much at least.

Going through a similar situation at the moment, so that's as much as i've got. Hope it helps though, and PM if you want to talk
J x