I’d tell him straight, if he tries to come in uninvited again you’re calling the police. That’s not on, even if you hadn’t dumped him. I dread to think what might have happened if he’d found you alone, rather than with your mum. Someone who does this isn’t a safe person to be around because they don’t respect your boundaries, and by extension your personal safety.