It may be better to keep away from him.
Why stir up the flames if you're only going to get burnt? It'll end up burning others, too.
You will find another man with whom you'll feel a strong attraction to, also. But, it'll never happen if you're too busy eyeballing this married one.
I may have misunderstood but it sounds like you got one leg in his bed already. Do the right thing and let it go.