Here are main points:

He was a good friend (or at least I thought so). Our friendship had started while he was in declining phase of his relationship which I came to know later about.

He was sex deprived and down the line wanted friends with benefits. He had also once told me that he loved me which I never believed. All I could see was he wants sex and I couldn't give in to that. (I'm a virgin and seems like grey asexual who don't feel need to masturbate leave urge for sex). I myself had tough past and am emotionally weak. I had told him all about that.

He couldn't get sex from me. Started avoiding me stating that he is harmful for me (yes, in a way that is true because if I'll give in to his sexual needs and he will move away, I'll be devastated. Probably would consider suicide even). Months passed. But then he returns, gives me happiest hours of my life in almost a decade. But, at end of them again he leaves me because I don't want to have sex. Tells me for my sake he is doing so.
In mean time his family gets bad news. I visited him because I wanted to be a friend he could rely on. Again same thing repeats. He tells me I shouldn't have visited him. Drops me home and things go down to sex again. Same story repeats. Leaves me wailing and I remember frustration on his face. He goes on long leave to be with his family. No contact, no news.
When he joins again, he comes to meet me. I tried to revive our friendship but he wouldn't respond to me. And then again he hangs out with me one night... I would say another time I felt little happy... and after that his needs outpowers everything else.
Again, I refuse. And after that everything just goes downhill.
I realised that in all this my feelings for him had grown much more than I wanted to admit. I feel happy with him else I'm always irritated. Tried to talk to him but he blocked me.
After some more months, again contacts but same story repeats. And now again I'm in his block list.

I understand that I can't give him sexual satisfaction even if I try to. Without commitment it is just not possible for me. I also know if he treated me well, I might have given in. He is first person I ever kissed and he knows that.
What I don't understand, if he ever cared for me, will he leave me to struggle on my own knowing very well that I'm not handling things well?

Then it happened that we met at a friends party and noticed how different he had always been with me. Also I interact with many common friends. And I find that he helps them in the same situations he has never helped me. It feels like he never cared for me though he claims whatever he is doing is for my best.

I really really want to flush him out of my life. But, I'm unable to. It feels like as if time is passing but I'm stationary.