I am exactly the same way. When people tell jokes or are sarcastic, I take what they say literally. In my case, I've wondered if it means I'm on the autism spectrum. I have a very high IQ, but interacting with people can be difficult for me if they aren't direct.