For me it's more that small talk takes way too much effort and energy, things I can't bring up when I feel depressed. I'm horrible at small talk anyway, and feeling down doesn't make it any better.