I do this all the time. I am afraid to answer because I think it is some disaster, but then I worry about what it was.
Usually I just let them leave messages, and when I think I can deal with it I listen to the message, if there is no message I figure someone called to chat.
I am a social worker so when I get calls at work it is usually a crisis, so when I am home, I don't like to talk on the phone, and often can't deal with it.