I think you just have to tell them that, that you are not interested (rather than "can't" which implies you have some other conflicting event scheduled). You could say something like "Nothing personal but I just don't like to mix my work life with my personal life".