Oh dec/trm,
I'm so sorry this happened to you.
Dealing with the "system" has let me down on more than one occasion. But there are some shining stars with genuine empathy who've helped me, so I don't want you to think it's all bad.
1) Are there any peer support group where you are who could advocate for you?
2) Might you consider speaking to your wife about this and perhaps showing her the letter? That might help you feel less-stressed in the long-run, no?
3) It might be worth it to respond to the letter to counter their points in a professional manner and send it registered mail.
4) Do you have a pdoc and a family doc? Would you discuss this with them to ask for some guidance and advocacy?
You met the wrong people... sadly... who were there to help you and they didn't. This has happened to me where I've been judged and alienated. So, it's normal to feel threatened and stressed. You are not alone.
I think the best route would be to set the record straight. The sooner the better so you can lay this to rest and have access to the services you need and are entitled to.