I would most definitely voice my objections. Taking on your ex as a client is really not ethical, particularly since he had already said he would not do so.