Yeah that's really weird. I would not have participated either, especially if the boss isn't that respectful. Sounds like your coworkers are very hypocritical and like to gossip to be honest. And yeah, I would not want to have some event you normally would have with friends and family at work. I like to keep work and social life separate, as I'm sure most people prefer as well.