If the tables were turned and you were making comments about good looking guys you saw, would he be ok with that? I think its about mutual respect and if he knows it bothers you, then he can tone it down.
Ofcourse, we all look at others, but we don't all comment out loud-- thoughts that hurt feelings of ones we care about.
I definitely have trust issues, and hope to correct my way of thinking, so this may be way off. Just the way i see it.