I don’t think it’s your fault. We know a lot of personal things about a lot of people at work as we are like a family but that doesn’t mean it’s an invitation to ask people out. Now you don’t work together so if you were both single it would be fine for him to ask. But you both married what’s he thinking!