There is no reason in the world that your son needs a 'sleepover' with his visiting dad. Explain this to you son, that what his dad is proposing is inappropriate. And just say no. 1000 people would back you up. It is ridiculous & he shouldn't get away with it.
Dinner out? Yes. Movies? Great. Drop the boy off after a nice evening? Of course.
If he brings the girl that immediately changes the trip, & the dynamics. He sounds dopey enough not to realize this.

If it were me I would say, 'when you can find time to spend with your son, devoted/dedicated time, let us know'.
What he does on his own time is his business. This is your son's time.
When my estranged (or, er, strange) H wanted to take my son on a trip to visit an old gf, I said, "NO. Inappropriate. Not gonna happen." And it didn't.

Father/Son time should be that.
I agree with previous poster. Hold Your Ground.