Okay, so you may not like this but...Is that good for you? What happens when he wants to see someone else? You will end up in the same place you were in at the beginning of the week, freaking out and on the verge of doing something drastic. Regardless of whether it is now or later, things will not end well with him; he will not end up staying with you. You are putting off the inevitable.

I know it's hard to be without someone who makes you feel comfortable, but you have to find the respect for yourself to walk away from a situation which is only going to be harmful to you. Not to mention that, my dear, he is using you for sex because he realizes that he can. Don't let him.