I am not a lawyer but I would not define that as assault.

He is, however, very sneaky and manipulative.

Worse, his playing games is even affecting the baby and he might end up turning her against you.