I do a lot of volunteer work and it turns out there are still difficult people and situations to navigate even when you are working for free! There is only a certain level of negativity I will put up with in any volunteer position because I am not being paid. If you decide to leave now, it may be obvious why you are quitting but does that really matter?