You post is confusing to me re/all the different men, but my concern is for the children. I think it is selfish of you to insert yourself into their lives, and I can't figure out, for the life of me, why the mother still allows you around them!
The decent thing to do would be to remove yourself from them and any male family members in their presence.
Period.
Seeker