If you want him off your back, you can simply say that you don't want the kids. Tell him he can have them. I guarantee you that he does not want those kids, and niether does his wife. He will try to play chicken, but don't be the one who flinches first.