I didn't see so much if a problem with it until I visualised myself saying "your looking very manly" to a male colleague or friend or anyone actually. I just wouldn't do it for it implies that at all other times they do not look like the sex that they actually are, and to me that would be an insult.