So this is what I messaged him, it's polite to the point but not rude in any way :Hey I am uncomfortable with the idea of your ex possibly coming round and having a sleepover. I know it's your house and all that but I don't think its appropriate. I completely get that you have friends who are ex's and I'd never stop you from talking to anyone. It's not my right to do so and it would make me feel bad as a person. I'd be totally cool if Stacie from work came over, cuz that would be really awesome. I hope you understand. ��