Personally I think that if he wants to be with you, he needs to definitely not be staying at his ex's house.

Have you told him yet what works for you? If so, what was his response?

Sounds like you are having to do most of the emotional work, ideally he would be doing what is needed to help you feel secure and safe in his love. He needs to invite you fully into his life.