He is an adult.I would tell husband,draw some boundary lines for my protection,or choose.I would have an alternative place to go to.If a man will not stand up and say to another male,you may not,under any condition speak to a woman,my lover/wife, this way,or anything on your behalf,what does it say about your safety and relationship?Where is your security emotional and physical?