Seems like your aquaintances enjoy spending time with you and agree to volunteer at the garden in order to do so. That's why they suggest alternate plans when it rains...

Maybe these people consider you more than an aquaintance or are trying to develop friendships with you.

I'm sure they'll get the message that you're not interested in friendship if you keep declining. Hopefully they don't feel slighted and won't stop visiting your garden because of it.