Gleak
I can't really tell you what to do because I don't know your wife or your relationship and I am not any type of expert. I do feel that honesty is the best policy and if it were me, I would tell my spouse my feelings on this. It doesn't have to be an accusation. It can be just a discussion about trust issues relating to you. This will take the pressure off.
Hope this helps a little.