I don't think there's anything wrong with saying "that's personal" or "I don't wish to disclose that". If he gets offended that's his problem.