I would run him away in that there is nothing, even with a "friendship" that seems to be working for you? He shows up when he wants, everything is on his schedule? If you do not want casual sex with him (still on his schedule), it does not sound like he is trying for anything more in that he hasn't asked you out or formally put anything into words or asked you what you want, etc.? He sounds confusing and I don't like feeling confused.