I wear cotton, wool, and leather shoes all the time while flying. The only nylon I own is life jackets, and immersion suits, which I do wear while flying. It's a compromise - but always cotton/wool underneath! I'm also a volunteer firefighter, so I could be getting into turnout gear on short notice - I'm hardly going to wear nylon/polyester under that!
While flight testing for a manufacturer, they insisted I wear a nomex flight suit, which was fine with me. I may go looking for one for other flight testing I do. It would look odd getting out of the my 150 following "regular" flying, but, so does the immersion suit!