Well, it's bothered you enough to re-post, so that is saying something pretty heavy.
Are you still afraid of him? Are you afraid of talking to him about it because of how he might react, especially if he says he doesn't remember it?
If you are afraid of opening an old wound, maybe writing him a letter about how you felt would make you feel better. You don't have to send it, but maybe it will ease your mind to get it out and forgive him for it.
I've been sexually assaulted, and as part of my therapy I had to write a letter to the guy whose name I didn't and will never know and forgive myself and him for what happened as a means to try to release the pain.

It's not the be all end all, and not the magic answer, but it may help you with getting over it by forgiving yourself and him for what happened. Maybe one day you will feel strong enough to tell him what happened, and that you're telling him because of the fear you felt that day but you aren't trying to hurt him by letting him know how you felt.
Sometimes people can surprise you. But I suggest the letter strongly, and maybe when you feel strong enough you could read it to him.