Hmmm.
Visit him, have amazing sex and be devastated and heartbroken when he doesn't change his mind...which he won't.
He doesn't need to he is still getting what he wants from you without the hassles of being in a relationship.

Or avoid all the hassle, heartbreak and lying to people who actually care about you by telling him to stuff it and staying home.
If he wants sex that badly he can come to you. And she'll out for a hotel.

Honestly I think this is a bad idea. I am all for free love and sexual expression, but that isn't what this is about.
This is you hoping beyond hope he will see you and remember what you had.

Only problem is, what you remember and what he remembers are quite different things.

And no I don't understand why you need to be a secret, except I think even he knows this behaviour is pretty low, which is why he would hide it.
Or is there another girl?

If you really need to do this to finally r get him out of your system then fair enough, but I wouldn't lie about it.
I would be totally honest and say your going to see him for a day or two to talk things over.

It's really important for safety if nothing else, thAt someone knows where you are.
And if he was any kind of man he wouldn't be encouraging you to lie, or trying to hide you from people regardless of wether your together or not.

Anyway this is just my opinion, nothing more.

You will do what you feel is right for you in the end.

Just be safe.