Just be honest. Say you do not wish to be rude, but you would prefer not to talk at the moment. Some people will be offended, but at least you will not be a liar.