Your joke to the male would be off putting to me, if I were a female in that class. So, you've ruffled some feathers amongst the women by joking with one of the men. Since you like the class, I would just keep going to class but I would be careful about what I said to people, women AND men. Just be yourself, but don't make those types of comments and don't flirt. It's just a class - there are other ways to meet people and to make friends. You can certainly start over somewhere else and try again in another class or group activity. Live and learn!