I think unless your life depends on it or the person is really dangerous, otherwise it is always better to speak up above all in relationships.