Totally agree: Indoors = hat off.
Outdoors = hat on.
Too many people seem to be embarrassed about wearing hats outside of the military confines. If it bothers you, wear civvies, you're more than entitled to do so, but please, please look smart when in the civvy world as it does us all no favours if you don't.