It sounds like you could use a good counselor skilled in child developmental issues to help out with this. It sounds good to give the child access but the interaction is clearly not healthy/positive----I think it would be best if you talked with the child to find out how they feel about the interactions and take the phone---having the phone puts the child in the position of having to deal with issues well beyond their years & may put them at risk.
You are trying so hard to be "fair" (I think) that you may not be being a wise parent. ((hug)) tough situation.