I can relate. I think at least part of it (for me at least) is that you literally can't relate because you're feeling good and seeing things positively. For me when I was hypomanic, it was like I'd look at problems people had and feel like they weren't a big deal, not in a mean way but in a, "nooo, can't you see? Everything's fine don't worry let's go be happy woowooo!" sorta way.
Then when I'm depressed, I perceive things more negatively and assume all other people are feeling/seeing things the same way I am (or worse), and think that even a small problem must be really painful for them. I think it's kind of like how people say that people who've been depressed are often the ones who are most empathetic and don't want to see others hurting because they know how it feels... When you're manic, it's kinda almost the opposite because everything's good and you almost forget how ****** things can feel.