I'm with gardengirl. Talk to him about it. That sort of comment is out of line, and if it makes you uncomfortable, you shouldn't have to put up with it.