I can't say that I support torturing anyone. Not even abusers. I think you are right in saying that some people are mentally incapable of understanding the harm they have caused to others. But your brother was only a child too.
Ultimately we can't make any one person do, say, believe, admit to, or take ownership of anything. Not even with torture. Torture doesn't heal anything. It only creates more harm, more pain.