I worry about it too and get over the issue by wearung exactly the same every week. I have a different t shirt on under my fleece but apart from that it is always the same. Problem is as you say, summer is coming up and it is getting too hot to wear the fleece. I feel too exposed in just a t shirt though. I know I am going to have to do something soon, I just don't know what.