Hi zekey,
I was only on lithium for a couple of months to get me through a particularly stressful period at school, so when I came off of it I was all right. (That being said, when I came off the lithium I was also finished university and heading home to a relatively stress-free environment, so I had none of the triggers you're describing.) I think if you've been on lithium for a long time you can experience a variety of side effects as you come down. For what it's worth, I've come off of other antidepressant medications in the past and that HAS made me feel worse. Have you told your doctor about how you're feeling? Regardless of whether it's the lithium or something else, you should probably let him/her know so you can figure out how to deal with it together.
Best of luck.