Why not tell him his worried tone is so similar to an angry tone that you find it unsettling / upsetting?

Only way to iron this out is to communicate directly and effectively.