Yes, my friend it IS emotional abuse. And many times it's worse than physical abuse because it lasts longer. Physical abuse heals in time. But emotional abuse lasts longer because it takes mental issues longer to heal.
I'm surprised you stayed too after *2* affairs. One I can see forgiving, but 2 I would NOT have forgiven. But I'm not you and I cannot judge.

It just seems that once a cheater, always a cheater. They never seem to stop once they get started.

And of course YOU are always the one they blame for their affairs. It's never THEIR fault. They can't take responsibility for anything.
I too was a victim of emotional/mental abuse for 26 years and I finally got divorced. I stayed for the sake of the kids, which was ridiculous. Kids are happy in a happy home -- and we certainly didn't have one. They would have been better off if we got divorced. I would have been happy -- and they would have been too because it would have been a happy home! But I waited til the youngest was 18 to divorce. And it DID effect them.

They've forgiven me and know why I did what I did - but they wish I'd done it sooner.
I can't tell you what to do but if you're miserable, life is too short to live like that. If he won't go to counseling, you haven't go many choices left. TRY and see if he will go to marriage counseling. He really needs it. Perhaps the marriage can be saved. But if not -- do what you have to do. God bless and please take care. Hugs, Lee