It's only as weird as you make it to be. Go with it, laugh at the situation ("um, is this a ladies only class?" or "Woohoo! I like my odds! "

Swimming is an important skill, and once you get past it, you'll forget about the male female thing. We're all just humans.