You are allowed to just say no. You do not have to give them consent to speak to each other.