I would seek a second opinion, just to help you make a more informed decision. Ultimately it's your decision whether or not you take them. Keep in mind though that today's antipsychotics can also have antidepressant qualities, so your doctor may have other reasons for having you on them than just for psychosis or mania.

Of course, if your sense is that your current pdoc does not respect your opinions and is not treating you as a partner in your treatment, then it may be time to find one who does.