If any particular type of apparel is chosen because it is best suited for the use/operation, then that is what to wear. I do try to wear shorter jackets while flying. I once had to fly my 150 in full fire department turnout gear - not good at all! (though if I'd crashed, I would not have burned!)
Unusual Attitude mentions pride and enthusiasm. Certainly excellent attributes for our chosen pastime. If you're a member of a flying club or school, and they have a nice shirt with the logo, by all means... If you need a "flying" jacket, because you fly in the cold, that sounds like a good idea to me. If you stop by the mall on the way home wearing it, and I ask you if you're a pilot, with pride, say Yes!
But, the coolest pilots I know, dozens of them, dress like regular people you'd meet on the street, their pride is inside, not so much in what they wear.