I'm sorry you're in this situation, it sounds really difficult. Try not to be too hard on yourself though; there are two of you involved in this, he is responsible too and it sounds like he's being quite unfair on you. Would it be worth maybe having a chat with him and letting him know you need some space and it would be best if you don't hang out for a while?