Have you checked or are you just taking his word for it? He sounds like he'd have little or nothing to lose by bluffing you.

For that matter, what would have happened if you'd required him to "formally" prove he had any right to be there before letting him back in?

I have no idea how landlord-tenant law works in your area. Would someone who had no proof they'd ever paid you any rent, and didn't have their own key unless they'd made an unauthorized copy, have any legal standing? Where I live, there are certain requirements a roommate has to meet before they become your tenant rather than just an overnight guest. I think some minimum length of continuous stay is among them. Some years ago, low-end residential hotels in a nearby city had a policy of not letting guests stay longer than 28 days just to make sure they didn't acquire the rights of longer-term tenants.

Good luck, NF -- but it sounds like even more than luck, you need some good legal advice.