When it comes to entering or leaving relationships, people tend to do what is good for themselves. So I agree that him moving away is not something he is doing mainly for your good. He is trying to ease his own frustration. But that's not really unfair.

He wants sex without commitment, while you don't. He's not willing to commit, and he's not willing to give up looking for sex when he is with you. So he's leaving. That doesn't prove he has not cared for you. But he has not cared to the extent and in the way that you are wanting. I understand that's very disappointing to you, since you seem to care very much for him. The two of you seem unable to get on the same page regarding sex and regarding commitment.

This hurt is still very fresh. In time, it won't seem as bad as it does now. Most single men are not too interested in being just friends with a woman they find sexually attractive. If having sex in a non-commited relationship is too dangerous for you because of the risk of being hurt, then you may be better off to let this relationship be over now.