Marital counseling is crucial in this situation. Your husband needs to be checked and corrected in how he's speaking to you and handling this situation. I'm not sure he's being intentionally manipulative, that's always a possibility of course, but regardless you need an impartial 3rd party in this situation.