It's ok to tell your Doctor you do or do not want to take certain medication.. I have a long list of medications that I refuse to take. My Pdoc is fine about it.
If you doctor doesn't want to work with YOU to find a solution that you are happy about , then it is time to find a new Pdoc no matter how hard it may be.
Good luck