Have you gone to an endocrinologist and had your testosterone checked? I'd start there if it's really bothersome.