We are all adults and as such we have our desires and temptations. He maybe a judge but he is also human. I have no idea what his marriage is like. They may very well have an open relationship. Or a ask don't tell relationship. Or it maybe as he said.
You have to do what you think is best for you. if you could except having just a friendship and live with that then whats wrong with that? I can't put a moral argument out there , Only you can decide what is right or wrong for yourself. I was very tempted when i was married and had a very real offer to have an affair. that i turned down. I was also once in an affair with a married woman when i was single. It as sweet and short in the end she chose to remain with her husband. Seeker, your a good woman with a tender heart. Be true to yourself in the end that is all any of us can do.