Well, either you talk to him and things get better or they don't. It doesn't sound like things will get worse. Bipolar disorder isn't a new topic with you. Perhaps you could suggest getting help with organizing the conference? Hell, maybe there is someone in your office who would like the opportunity for some additional responsibility. Thing is, if you don't talk to him, you are more likely going to end up on a leave, right? I suppose you do have the option to suffer through everything and tell your doc to go to hell regarding a leave but, compared to an uncomfortable conversation with the boss? Eh.

I have a terrible time saying no to people too especially my employer! Stresses me out but, stress is good for me sometimes.