I’m quite a private person so just reading this makes me feel irritated on your behalf.

I guess a lot depends upon who’s asking and why so there’s no broad response. If it’s benign I’d be gentler, especially as you say it’s a small town. Maybe smile and say “What? Working here? I’m too busy” if that sits comfortable, and move very swiftly onto something else.