He's not refusing to talk about it and he isn't abandoning you. He's simply saying this is something you two should talk about in session which seems reasonable. You can certainly cancel the session, but it won't resolve the issue; you will still be left feeling distraught. Sounds like you need to sit down with him and talk perhaps. Pulling away when you are upset seems to be reinforcing your belief that you are being turned on and hated, but he hasn't gone anywhere. He's just asking to wait to discuss this in person.