For example if you entered an Officer's office, the first thing you should do is salute the officer, but you can only do that when wearing your hat. So as far as I know hats should be worn indoors - end of.
Please tell me you are not including the RN in this blanket statement.
If you saluted on entering an officer's office in a ship or shore establishment, he would think that you were taking the proverbial!
The last one I saw (a Lt) was scruffy as hell, no hat in sight, shoes that had never seen a coat of polish and he was pretty hefty for his age (looked like 25ish).
If no "hat in sight", then quite possibly HMRC rather than RN. Incidentally, naval officers and ratings wear caps, not hats.
Jack