oh my, having seen the inside of quite a few hats I would suggest their removal would be a safety hazard. Not only are most of the details of their personal life scribbled in turbulent script into the linings but some important stuff too.. (or so they claim)
while I am on the subject...
Why do Barristers & Judges in many jurisdictions wear robes and horsehair wigs?
anonymity, sobriety and tradition. The underlying reason is of course that it was fashionable at the time of introduction, and has been unfashionable for so long that it has become custom.