It sounds like you have zero interest in romance with this guy. He has made it clear that he does have a romantic attachment to you. I think both of you are too old to be hanging around each other when you are not on the same page about what you want from each other. This kind of "friendship" has no future, IMHO. I think the two of you need to either agree to date in earnest, or go your separate ways. Keeping on the way you are is destined to lead to something ugly when one of you finds a real partner.