Well, I deal with slightly similar issues - I am physically very unattractive and am very well aware of it - it's not dysmorphia though, because it is not a misconception on my part. For me it would be very inappropriate to have a therapist (whether the therapist was male, female, ungendered, old, middle-aged, young, or other) compliment my body or my appearance in any way. Just as inappropriate as it would have been if I'd been good-looking.

I agree with your last statement, though - if you think it's inappropriate, then it is inappropriate for you. Doesn't mean that it is inappropriate for everybody, but you should be allowed to react in a way that is right for you.