It doesn't sound weird at all. They're both obviously concerned. I think the big difference is whether they're trying to make him better for him, or better for them. That's hard to work out.
When I was a teen, my dad was pushy about having a routine, staying active, and things like that, which was a bit too much, sometimes really upsetting, and not totally appropriate, but wasn't damaging. He was mad at the world, and it sometimes got him aggressive, but I could always stand up to him and make myself heard with him.
My mum pushed for what she could do to fix me, had no trust or faith in me whatsoever, and really undermined me, and sabotaged my life. Didn't actually want me to be better, just less problematic for her. She could not handle her own disappointment. I wouldn't have been able to say that at 15, but was always happier away from my mum. I would have said at the time that she was a good person, and I wasn't a good person. My dad always thought that way, and my brother still things that way, himself.
Just saying there is different ways of trying too hard, and it's not always the person that shouts the loudest that's dominating the situation.
It's good that your nephew wants to be closer to his little brother, and that he can trust you.
Positive reinforcement, and encouragement is less likely to backfire, than pointing out flaws...if there are any happy memories or happy times you can draw on? I guess it's also important to let your brother know how his behaviour makes you feel, and hurts his son. With PTSD he is more likely to feel attacked even when he's not being attacked, though. Even if his symptoms don't seem so intense, getting out of an abusive marriage is really dysphoric, and could be really hard for him to get stable after that. If his ex's behaviour has triggered that stress response, he could well be getting triggered often with his kids, too. Think there's any way of helping him shift his try-hard attitude to positive things?... getting him building stuff up, and maintaining things, rather than being in crisis management mode all the time?