I’d not worry about the father. Not sure why he takes her away but I suspect he is trying to get her off drugs. Clearly it’s not working. I don’t really understand how two of you still have custody of your children when you two are actively doing drugs.

Her father doesn’t have power to get you two divorced. It doesn’t matter what he says but he has power to report you two to authorities for drug use with young children at home.

I recommend rehab for both of you, see a doctor, therapist etc to get clean.