He's not right and you shouldn't feel guilty. You could try couples counseling if you want to stick it out or tell him that you've apologized and it's time to move on and you can't be with him if he continues to throw it in your face. Good luck.