Being in a uniformed emergency service i have to wear black trousers, white shirt with epaulette loops and a black clip on tie (because we aren't capable of doing knots). On a occasion when doing my PPL I'd go straight from work. I'd take my epaulettes off in the hope people wouldn't notice. My instructor, young enough to be my son (!) would be flying in jeans and tee shirt and I must admit to feeling a bit of a fraud when wandering round other club houses as people would aways assume our roles were the reverse of the truth.
(Cool story bro?)
I do always wear cotton or wool long sleeve shirts, trousers and undies when flying having seen the effect of burning synthetics on skin and hair (! again) at work...