Thanks for all your replies. I know you all are right, but it's difficult for me sometimes to feel self-confident and say "no" When people insist and maybe I get nervous, it's difficult for me to think fast to an answer.
But yes, if I meet him again, I think I'll have to be "strong"...though it's not very spontaneous for me and it won't be easy.