I talked with my pdoc about the phenomenon. He suggested that it could be that our brains are producing endorphins to deal with the stress. When those stop being produced we become susceptible to feelings of depression.

Since social experiences maybe rarer for us, it makes sense that we would be impacted more severely than others without anxiety or depression issues.