A doctor told me a long time ago - "Junia, any given situation can be seen in a hundred different ways, 99 of which are probably wrong." I've tried for a long time to see multiple possibilities in a situation, with varying success.

Here are some alternative possibilities.
Maybe he made a very poor joke
Maybe was was making a mediocre joke that would have been obvious if you could have seen his eyes.
Maybe he was upset with someone else at your facility and stupidly took it out on you.
Maybe you misheard him.
Maybe his stern face and tone were part of his intended, very poor joke.
Maybe he was speaking for the benefit of a nurse or other staff member who really was stalking him!

I do know what it's like to have that stomach-clench that lasts for years - I have one that goes back to 1987. But it can be really helpful to look again, and imagine a new interpretation. It is for me, anyway.