It's a slippery slope, because it's his ex girlfriend! And going to her house, and lying about it, is completely unacceptable!
Dirty texts and 2 hour visits to an exes home, sounds sketchy to me.
Perhaps, you are having a tough time learning to trust, because he's not actually doing anything to change his behavior?
Ultimatums are not the solution, but putting the ball in his court, to meet your relationships needs is!
""I am feeling insecure and vulnerable right now. I'm out of my comfort zone about this friendship. What can you do, to help me feel reassured that you are devoted to me, and that I am your top priority?"" (that's one way to express it...)
I'm sure, others here, will have some sage advice