It's a nasty situation.

He is not well and is not making rational decisions right now.

It's six of one hand half a dozen of the other.

You could stay away as he's ordered you to do. He seems really angry at this point in time. And he did tell you to leave ....

If you return, you could face his wrath again because of his mental instability right now.

You need to do what makes you and your baby safe right now. Whatever choice that may be.

Wishing you well.