It's difficult to stay in a relationship when you feel as though it's a temporary fix, believe me I would know. I think if this is how you feel and you feel relief every time he is gone, then maybe you should suggest some ways that could improve your relationship, and if that don't work or alter any of his annoying behaviors, then leave him. If he's not capable of changing small behaviors for you then it's not worth making an effort for him.

I hope all works out, please let us know how everything goes.