Dear Kotomi,
Why not tell them you're sick that day, that it's not infectious, but that because of it you can't go swimming? And busily volunteer to help in the kitchen or with toddlers or something else that will keep you in the house and not at the poolside. Always with a smile, of course!
Hope this helps!
Take care.