You got involved with a man that already has children, so he is always going to have some contact with their mother pretty much the rest of there lives as school graduations, weddings, grand kids are probably going to happen.
You can be angry about his ex, sure. But he choose to go back with her, his choice. I assume you have been able to forgive him for it???
Instead of looking at the children and being angry because you see their" mother" why not see it for what it is , they are your husbands children.
The fact is when familes divorce you are going to hear kids talk about times before the divorce, what trips were fun, funny stories or joking around ,etc, That's what kids do . You can't expect them to never talk about things prior to you and there father getting married.
If the children are not behaving when they are with you then "house rules" need to be discussed and agreed upon by both you and you husband about
all the children.
I think you could really benefit seeing a Therapist you have a lot of anger going on, Therapy can help you identify where its coming from and processing it in a healthy manner as to move forward in your life.
I'm sorry your going through such a hard time right now.
Welcome to PC