I think if you can't trust your ex, then he does not need to have a key either.
Some other arrangement needs to be made for everyone's safety and your peace of mind.