I have gone totally off mine I think 3 times (slow learner). I ended up in the same condition each time. I hit rock bottom. Each time was a different location and circumstances. Definitely no recommended.
You must remember that your treatment is for you by you. You are the driver of your therapy. I know sometimes it feels like you are just following Dr's commands, but you're really not. You are helping yourself by getting treatment and that is your decision!
You have the right to question Dr's suggestions and put in your own.