maybe he is able to be "younger" with you because he trusts you?
Or maybe that is his professional voice. I was raised with one. I can be sobbing my eyes out, or just awakened and I'll answer with my "receptionist voice"; the one where people can't tell the difference between my mom and I.
You might even make a recording of it and ask him if he notices the difference and what he thinks about it (if you think that would be ok), or just outright ask him if he notices he speaks different on the phone than with you.