this is weird, but I've had concerns about my dad cheating on my mom before. I'm too scared to confront them and ask my mom if dad did ever cheat on her...but it could all be in my head.

At therapy last week I learned that I really don't trust my dad because of something he did a few years ago that really violated my trust. So maybe my thoughts could be coming from this.

Don't know if this helps, but maybe did your wife do something else to you that would have really violated your trust of her? Maybe something else happened causing those feelings.