The problem with Hi-Vis Regulations
In defence of the CAA (although I normally prefer the other side of de fence), CAP 642 is absolutely, totally NOT responsible for the over-zealous interest in hi-Vis Jackets.
But, if it's in the official, CAA approved, Published and Annually (anally? -sic) Audited Aerodrome Manual for that particular airfield, then the Aerodrome Management have to enforce it - otherwise it's a black mark when they next get inspected.
If it's not in the aerodrome manual, you don't need it.
Why not just carry one in the back? Frankly, its just easier to put it on if someone asks you to.