wearing gold bars when you are not a professional is not cool, wearing a flight suit is ok as long as you are not covered in unit patches of units you were never a member of. But as an item of clothing that is both useful and can save your life, there is nothing wrong with wearing it.
I have and do wear it if I'm flying vintage or an aircraft that I've never flown before, I also wear leather boots and gloves. I always use a helmet if I'm flying an open cockpit.
My personal rule is safety before fashion, it's better to be alive and ugly than dead and beautiful.
Fats