No, you are not asking too much. Give him a date to get out, put it in writing, and schedule the lock change for the day after your deadline.

It may be worth contacting your local law enforcement to find out what you should do in this situation.