I don't know if this would work for your family, but we had our backdoor bolt lock replaced with a keypunch lock (non-electrical, just a 3 digit punch button code, but also a secure deadbolt). It was maybe $20 more than a regular bolt lock.

Teach your kids the code (my 10 year old really likes to open the door) and then there's no awkwardness with the ex. Of course, the kids may pass along the code to him or he may learn it from watching them open the lock. He still could theoretically pass along the code to said shady guy, but there's really no reason to because the code doesn't need to be "left" with someone.

I also feel safer somehow with my kid being able to get into the house if the door is locked, he doesn't need a key.

And you can change the passcode on the door lock anytime you want.

Anne