He may have read the text too quickly and just registered the part about "might call in if I don't feel well enough"---which if read too fasst makes it sound like you have a cold. No one would argue that you should still come in with a physical ailment.

It is worth asking him what it meant. That it hurt that he didn't urge you to come in, to welcome you.