I don't know if there's a right time to end a friendship, especially one as important as this one you described. It's certainly a very hard thing to do.
I relate so much to what you wrote, I've been in the same situation some time ago. I just couldn't let go of the person who was my best friend at the time. Eventually I had to, because he wasn't interested in maintaining our friendship any more. I still miss him very much and I still think of him every day, even though I haven't heard from him in more than a year.
I don't know why I'm posting this, since I have no advice. I guess I just wanted to let you know that you're not alone... and that I understand how special some friendships are.