I think you have to draw the line somewhere. You should not take all the abuse, its unhealthy. I suggest that you and him try marriage counselling, so you can air out all of your feelings and he can too.