I'd love to agree with you - as this is my personal philosophy most of the time.
The reality is quite different to be honest. I work across various fields, and in particular know world leading scientists who would never be seen dressed smartly, and superb pilots who are rarely seen in anything but jeans and a t-shirt. (Not to mention a few very smartly dressed people who were very poor at their jobs.)
They aren't the same. I choose to dress smartly and appropriately for the environment I'm in, but I've learned not to judge other people by my own standards on that point.