Have you thought about what upsets you when you leave them? Being alone with your thoughts? Missing them? (you dont have to answer, just trying to allow you to think!)
I hope you feel better soon!