I would not be willing to leave a child with someone that is going to yell if a baby won’t stop crying , babies will do that , they are babies for gods sake.

I’d find other child care and go to couples therapy and he needs his own Therapist to help him with anger issues.