When people start prying around into something I think is none of their business, I change the subject or excuse myself to "go to the bathroom" or whatever. No explanation, no apologies ... just not something I want to discuss or explain. It feels a little awkward for a moment and then we are off on a more pleasant topic. Case closed and I've kept my dignity.