I would just say that this is your usual look or that you were just thinking or something like that. I dont think it's nice of anyone to point out how scared we might be. Just make light of it and change the conversation. Some people say I look sad, I just tell them, that's me! People need to accept you for who you are. It depends how comfortable you are with the person in what you want to respond with. Either way I would move the conversation on to something else.