I would be very concerned about having a drug addict around my child at all. And there must be more to his losing contact with his children than just drugs. I'm sorry but to me there is something not quite right with this.

Why is your influence not adequate for the court to allow contact? You are an adult with custody of your own child. His father not liking you? There has to be a reason for that especially if he shows it overtly no matter who is there. I know I'm just echoing what Perna has said but they are very valid points.