I am so sorry that you are in this position. Not only did my father cheat on my mom. He even had the audacity to take my brother and I along on one of his visits with his mistress. It hurt me terribly. Mom divorced him when I was 15, finally! Life was actually a little better without him. However, my mother turned to physically and emotionally abusing me as a way to cope, I guess, with all the stress. "If it isn't one thing it's another"...became my philosophy for the longest time.
I would definitely confront him..that is...if he isn't the abusive father that I had the misfortune to have had. Let him know how deeply this affects you. You are certainly entitled!!
Good luck.and have a Merry Christmas for yourself!!