I have been married for 28 years to the wrong person. Everything he does annoys me as well. The positive part is that I am very adept at self soothing, and can go off on my own and do things that are interesting to me. IMHO, having an affair would compound your problems. Besides, HE is annoying as well, you just don't know it yet.