I had one of those relationships when I was in college, he'd call me once or twice a school year and we'd do something fantastic with his friends and have a great time but nothing ever "happened" otherwise!

I think you have to discuss with him what his ideas/plans are, how he feels about the two of you together; we could guess all day and not get it right?