I would not think that you would have the right to do that, or at least not ethically speaking. I have no idea about legally because the law is often so screwed up anyway. They are his children as much as they are yours. He has as much right to spend time with them as you do.

If you have some evidence that he would be exposing them to any actual harm, then you would have a case to deny him seeing them. However, having lots of "friends" and allowing the kids to meet those friends sometimes is not harm, and I can't imagine it being deemed as such by any rational judge.