I think putting yourself first is not selfish, it's reasonable and a must do thing. What I read here:
1) he's pressuring you
2) talking to him causes panick attacks
3) he said he wants to hate you (?!)

You have to think of what's best for you, of what you want, and not how to make him feel happy. From what you've said here, it doesn't look like he's thinking how to make you happy.