I also feel sad that he can be happy while I'm not.

I'm also worried that this concern and time spent working on our relationship is taking away from my feeling of peace and livelihood.

I should probably not stay with him, for his and my sake. For his sake because he'd have an unhappy girlfriend who doesn't like him for how he naturally is...