"children" in their 20's - do you allow them key to your house once they have moved out?

Thought that I wanted relaxed "feeling" in my home, where my adult "child" could drop by and "pop" in - even if I wasn't home.

I've very much changed my mind, due to certain behaviors on my son's part - just in general a lack of respect for my property. It is very disruptive to me...

What is your opinion???
Any ideas on how to change the free access without totally
alienating him??

Thanks..