I agree, assertivity in this case is the best solution. I absolutely understand and get how difficult it can be because I'm shy as well and I find it difficult to say no. But perhaps try to view this as an exercise. When you look at the situation, although it looks overwhelming, there's really not much to lose and this isn't a really serious issue in you life, is it? I mean you're not "sinning" in any way, this person is not your family, and you're not doing anything wrong or illegal. So take this like a game, to see how you can practise your communication skills and if something goes wrong nothing that bad is going to happen, right?
At least this is what helps me when I'm anxious about social situations.

Keep us updated.