If you are not comfortable; you are not comfortable! How you feel is not a function of anyone else and their opinions about how you should feel.

I would just state categorically to your ex that you do not want the keys to your house left with that person, not accidentally, not intentionally, not at all. I would remind your ex of this person's past behavior toward you and that you do not like this person and explain you are angry that your ex left your house keys with this person and if there are problems in the future, he will be paying to help solve them.

Don't even discuss how you "feel" or whether you are "over-reacting"; tell him you are merely responding to his irresponsible behavior, not reacting to anything.