For the women who always have to dress in the latest fashions and wear make up all the time, it's probably that (they think) your casualness subtracts from their stylishness. They don't want to be seen without, so it makes them uncomfortable to see someone without. I wouldn't wear sweats to a job interview, but I'd rather be comfortable than not.