Don't tell him. It would be awkward. What do you expect to get out of it?
A relationship with a patient is unethical.

It's one thing to talk about transference with a therapist but it's just going to make it weird with your primary care doc.