Careless and not thought out statements is one thing, but his reasoning about a sense of agency seems way out of line. For me trust would be completely lost and I'd be looking for a new T. I don't think you did anything wrong at all, letting him have it for a careless comment like that.