I used to have a poster when I was a teenager that said "If you love something set it free, if it comes back to you it's yours, if it does not, it never was."

It sounds very much like this relationship has taken a turn toward toxic. One of you has to care enough about yourself or the other person to just cut off ties. If he knows how much it upsets you having him in your life (on his terms) and does not care enough about your mental health to say good bye, I wonder about his character.

You can put it in a box and wrap it with a pretty bow, but what it sound like to me is he wants the benefits of your relationship without the drawbacks. He's free to dabble with you then do whatever else he wants... which would be fine if you were both in a place to handle that type of relationship. It doesn't sound like you are.