Hi -- I'm the same way. I would suggest not sleeping with him, no matter what as a start.

Any chance you can move someplace further away, to reduce visitation?

Good luck with this. It's a sticky one.