What has led him to this conclusion? That you want comfort and support from him right now?

I don't think there has to be a physical attraction, but I'd imagine there would need to be some sort of sexual or romantic thoughts about him. Even if you were having those thoughts and feelings, is there something to be gained by calling it out? Is he trying to blame your recent ruptures on erotic transference instead of on his inability to listen to what you're saying and provide support right now?