I guess I agree with OlivaB. I have CSA issues and my male therapist sometimes does or says things that create issues between us, either because he's just screwed up or because I've misinterpreted him due to my own issues. In general, it has been helpful for me--not what he said, or my getting angry, but the hashing out process between us as we struggle to communicate and reconnect. Therapists make mistakes --this seems to me to be almost inevitable-- and to me what matters is how they handle the mistakes going forward.

Only you can know if your therapist is basically an empathetic, understanding person who inexplicably said something stupid and deeply regrets it and is trying to repair the connection or if he is an idiot and truly secretly blames you for an assault that could not be your fault. Didn't you say you've been seeing him for 18 months? Is he generally reliable and trustworthy? Is what he said out of character for him?

The "most important work" thing he said is striking. This may be the first real rupture you've had. Ruptures, and their repair, offer the opportunity for real personal change and growth (for both parties in the relationship), but it can be excruciating, and sometimes the therapeutic relationship doesn't make it. Whatever you ultimately decide, if you do decide to ditch him, I hope you can end things with him in a way that honors the work you've done so far and what you have shared together up until this point. I am sorry this is so hard and I'm sorry for what he said, and for what happened to you.