Being married does not mean the other person "owns" you and do whatever they want to. You need to let him know that this is a problem and it needs to be addressed.. So what if he gets offended.

Good Luck