Just leave him.

There are so many red flags in this relationship from the very beginning. Don't do your child the disservice of having such a man in their life.