I do the same thing, freak out if my boyfriend invites someone to come into the house without picking it up first (ha) or letting me know so I can clean. I don't like people coming into my house in general and I hate when they come in just on the spur of the moment. My house is always trashed.
I really think this is something wrong with ME that I feel like this and it is not fair to treat my boyfriend this way or to hold him to these standards that I have and he doesn't--- so if you figure out how to stop feeling like this, let me know