My dear Shangrala,
I can understand your pain....This is not an easy choice to make, but you have all reasons to leave him alone....If he was good to you at the first place, then you would have not gone through such a pain and sorrow right now....
You are not a bad guy, but you are if you think that way....just get that off from your mind...you got to do what is good for you....
You will hurt yourself more by staying....so, who is more important YOU or HIM?
besides, you can always come back....so, just do what you feel to do and don't think about him and his feelings too much...you are not responsible for others....and he will be fine...
let us know how you do, I wish you the best....
with love

Marjan