The big question for your husband to answer is, "just what part of 'forsaking all others' did you not understand when we said our marriage vows?"
It doesn't matter whether he's "hosting" men or women, the point is that he's decided on his own to break his marriage vows with you. And he's apparently doing (or at least soliciting) this in your home, with your child present. His behavior is outrageous, and extremely disrespectful of you, your health, your marriage, and your child.
I'm with the others -- if he were mine, he'd be out of the house NOW, with or without his bags packed. Not all the good ones are taken, not by a long shot. There's no reason to stay with a creep like that.