Trust your instincts on this one. It makes perfect sense to me!
I would have the lock tumblers changed immediately because it's so important. It is your place and you have the right to feel secure there, and to make sure your kids are secure, too. Easier to change the lock tumblers now that to have regrets later.

I also would definitely raise the issue with your ex and calmly explain the reasons why he may not give the key to anyone else without your permission and knowledge, that it is for the safety of the children.