I don't wear a flight suit (I did during Phase 1, though - and if I could find one that I could easily wear & take off over my normal flying gear, I'd probably still do so!) but I certainly do wear a tactical-style vest filled with survival goodies, (first aid kit, PLB, space blanket, etc) when I'm flying anything more than just circuits. The usual explanation for anyone curious enough to ask is something like "I'm flying in a plane I built in my back shed! If it goes to ****, I'd like to tip the odds in my favour..."
To my mind, it's one of those things you never need until you do, like car insurance, you're free to not have it, but you'd better hope you don't crash into a BMW. So if the worst that can happen is someone having a chuckle at my expense online or at the bowser, so what?