I think if your therapist is recommending digging deeper into your childhood, he is probably seeing something you’re not - either through the way you interact with him or through your current struggles. If you trust him, I’d just go with it.
With that said, I also think it’s normal and totally okay to not want to dig back and re-color your childhood and the people in it. I look at a specific person in my childhood very differently after bringing her into therapy discussions, and in many ways I wish I had left that part of my past alone. In the end, we can all be stripped down as very imperfect beings.