I think you should stand your ground. I'm concerned that he's violating other people's boundaries as well. If I worked in a hotel and a guest answered the door naked, I would be uncomfortable.