I think it just depends how much it bothers you.

My last male manager used to call me 'darling' and hug me - I didn't like it but not so much I wanted to make a scene. Lots of my female co-workers call me hun or similar and that doesn't bother me at all - it's more bothersome to me if it's cross sex.

Like Wovengalaxy says, it's completely your own preference and up to you whether you say anything or not.