Just say no. The guilt you would feel even if it never went past talking would be enough to tear you apart even further.

If this doctor had any morals or even just a single strand he would never nor should he ever act this way. He's using his position as a Doctor to probably do this to loads of women, no offense but I doubt you are the only woman he has done this too.

You said he just isn't your type.

As Tisha said spice up your marriage, he is a guy that is your type or you would have never married.