This happens to me a lot as well, with friends and at work. . I try to brush it off as bad memory. Everyone knows I am really really bad at remembering things!
I wish I knew how to be better at it but I don't yet.