Yeah, I don't understand the point of him saying that to you either. But the fact that you don't want to be with him anymore is the main point. Don't enter into arguments with him and if you like, just tell him to stop bothering you.