Is it love that you feel for this man, or the desire to be in a loving relationship that you love?
If he's physically abusive, never mind that verbally--words sting and damage, how can that be someone you can honestly look at a feel 'love' for?
He's miserable, yet refuses to leave? But he's having affairs. Whatever you may have done during the first two years of your marriage is irrelevant and doesn't justify how he treats you, right here, right now.
You 'grew up', after whatever it was that you did, the first two years. You are a mom now. Is being beaten something that you want your child to witness time and again? And even if they don't 'see it', they sense it, can hear things that may not make sense to them now, but it's creating a sense of 'normalcy' for them when people stay in relationships that are abusive.
Do you currently have a therapist? Do you know why people stay in abusive marriages? What brings the abused to the relationship and what brings the abuser to be abusive?