I understand why you'd feel guilty, but it's not your fault. Perhaps try to discuss this with him? Ask him to be honest and see what you can do. It might only be your imagination, as well.