Gee it's be nice to get an honest answer from him about it, but you do need to be assertive and look after yourself and if that means asking him to leave, so be it.