I just step away for a bit...though usually i say something first these days. I think people deserve to know if you're unhappy with them. Even if they could care less i think honesty is always the best policy, it also means you get to step aside with a clear conscience.