There is a saying, "what others think about me is none of my business."

But you are taking it a step further in the opposite direction with "what i think others think of me IS my business."

You are projecting these negative thoughts onto Bob. I cannot really tell whether you blame yourself or your wife for needing IVF. You thought you could keep this secret from your own heart.