It's all very well saying "wear the kit or die". IMHO it's not that simple for the average GA pilot. Take the example of flying across some cold water near the UK on a warm spring day, if one wears gear suitable for keeping oneself alive in the VERY unlikely event of ending up in the water you get so hot it probably impairs judgement and flying ability so you are more likely to end up making a stupid mistake and getting wet.
Ergonomics and comfort do matter for safety.