Calm down.

First of all, this is between your Mom and your Dad. I know you feel protective of your Mom, but if he's treating her badly, don't you think SHE should say something about it to him? She's the one being treated so badly. Or is she too meek and mild to speak up? Is she afraid of him? If she is, WHY is she? Has he ever hit her? Do you think he ever would?
Again, their marital problems are between them. Your Mom doesn't HAVE to put up with it, but she does. Is this the first time he's ever acted like this? If it is,, I"m sure your Mom figures that this will just blow over, and the affair will end. Many times they do -- she probably figures he will stop the nonsense and things will go back to normal. I'm sure that's what she wants.
In the meantime, don't complicate things for her by snapping at your Dad and making him even more angry. That will just upset your Mother. I know you're angry, but you don't even know what the truth IS. Just be respectful -- if you don't want to talk to him, just keep it short, but RESPECTFUL. That way you're not upsetting your Mom, and it's helping to keep peace in the family.
I wish you the very best. I hope things work out for everyone. God bless and please take care -- and try to stay calm, as hard as that may be. Hugs, Lee