I don't think you are wrong for thinking that way.

Let them know that the Fentanyl is an absolute No-Go for you.

I also think doctors should neither ignore nor diminish a patient's input regarding their care.