I really want to respond, but my brain is very fuzzy at the moment and don't think I'd make much sense...

I would suggest asking your t for clarification on his intent with the statement he uses to challenge your "I don't know"... that may be a good place to start a dialogue around your feelings about it. He may be trying to coax you into remembering something. I have had T's do that when they think I should be connecting something. At times like that, I tend to get frustrated when I don't actually have a connection...
I would also suggest perhaps explaining to him that you truly don't know, and you are not just burying something you can't say with that blanket statement...

Good luck. I will try to check back again when my brain isn't so foggy...