Using something shocking or absurd is a typical thing in humour. That is why it is a joke. It was definitely not about you. However, using "magic mushrooms" to someone who actually likes fungi is lame. You didn't get it was funny, but you also didn't realize it was typical lame humour. That's okay. Eventually you learn these stupid categories. In your case, maybe the guy really did grow magic mushrooms (but more likely something for research, no?), and even so the idea of "magic mushrooms" is funny to people - especially when they mention it to someone serious and possibly square-looking, such as me or maybe you.

One time I was in a writing class, and a girl wrote something I thought was utterly hilarious. A "space opera" - I thought it was a parody. No, she was serious, and it was just awful. I was laughing my head off until I saw her angry face and everyone else's serious face. This is why "getting the joke" can be dangerous, too. For people like me, so many things people do are unthinkable or ridiculous, but people are serious about them.