When the CAA still had a regional office at Kidlington, a chum used to pop in to Brize now and again to do the odd instructor revalidation session on his way home.
The CAA's uniform was exactly the same as HMRC - and the flying club was right next to the Air Movements cargo hangar.....
One day my chum arrived in his CAA uniform with 4 gold bars - and the 'movers' went into a flat spin - scurrying around moving various boxes out into sheds etc. It was clear that they thought they were about to be rummaged by the Heathrow customs team.
He saw this and suggested that he'd better take his rank slides off next time. But we told him not to - the sight of him in his uniform would discourage any dubious activity amongst the movers.
Personally I think that people who have a problem about the wearing of insignia by others have more of a problem with themselves.