I had that happen with each of my therapists at one point or another. I don't think it is a matter of it be "unfair" to feel upset or let down; that seems to be a very common reaction.
My therapists always made sure I had a back-up person/plan while they were gone. I never contacted my therapists while they were away. I either had another therapist or a pdoc I could contact if needed. But honestly, I was usually fine.
In the cases where it was an unplanned emergency, I completely understood their need to be away, but again, don't beat yourself up if you are feeling upset about it - that is pretty usual. I think because I have a large family and have dealt with many health emergencies myself, it didn't bother me too much. Been there myself, and mostly I just feel bad for anyone having to deal with sudden family emergencies; it's stressful.