I do the same thing with the elevator buttons. We are unusual in doing that, but I don't think we are excessively cautious. We just have thought it through more than most have. Years ago, it was more common for people to where gloves, even in good weather. I think that came, at least partially, out of a desire to avoid nasty things in the environment.

Echoes is right. Contact with bacteria is part of normal life. Actually, we need it. It is the good bacteria on us, and inside of us, that helps crowd out the harmful bacteria. When a baby is newly born, hospital staff encourage skin-to-skin contact between baby and mother. That is partially to allow the baby to pick up normal skin bacteria.