if your ex hasnt seen this guy for 2 yrs he doesnt even know him now!
i'd definitely have the locks reset & ask ex to pay cost, explain the man is really a stranger now & children + home are at risk, as well as yourself.
You have good gut reactions, Soup. listen to them now. i think you're being realistic.

I wouldnt bother trying to make ex feel guilty. wasted effort, most likely. "just the facts, Soup!" give him nothing to aim at.

& give one of your kids the key. at any age, s/he'd be more responsible.