Maybe you can call him and tell him that you changed your mind and would like to come back. When you see him, you could let him know that you got angry with him and didn't feel like seeing him again. You could say, "When I speak negatively about myself, you tend not to say anything. When you don't say anything, I feel hurt because it makes me feel like you agree with my negative statements." How does that sound?