I see it as a kind of sexual abuse. How old are you? Can you get away? If you are still in school, I think you should ask a counselor for advice. Let us know how things go--and do not give in to him. Have you told him the answer will always be "no"? Maybe you can tape him or even get a video of him in action.

I can understand how your grandmother doesn't want to believe it all. It could be that he's getting senile and losing all his inhibitions. Could you fight him off if it came to that? If possible, try not to be alone with him. Okay?