Nothing else matters except that he is married. Not for him or his wife but for you. You do not want to fall for a married man, trust me!! There is nothing but horrible pain and anguish in it for you. You will get hurt beyond belief and that is a guarantee. Just don't go there and spare yourself this torment...I've experienced it, so I know.

I can tell from the things you have said that the morals surrounding it won't stop you but do it for your heart. It is not worth it by any means. Protect yourself and let the crush run its course without acting on it.