You would be wrong. It is the headquarters of the UK CAA. A lot of people wear suits there. Nobody will give you any strange looks (unless it is your birthday suit) but there is no requirement to wear anything other than what you are comfortable with. For a good portion of the time you are there, you will be in your underwear and a bathrobe.