I don't know that I am completely convinced that this started AFTER your affair. He was distant and unresponsive to you before the affair, it makes me think that he was getting his "needs" met elsewhere which is why you were being ignored.