If you take the trouble to read the appropriate CAP you'll see that HVV is mandated for pushback crews and snow clearance operations.
The remainder is down to the aerodrome management.
The prime pupose of HVV is not to stop you being run over by an aeroplane. They are generally big noisy slow-moving things with flashing lights on them. The dangerous ones are the ramp vehicles that come whizzing out of the gloom beneath the terminal driven by someone wearing the aforementiond HVV that reflects in his windscreen and makes it difficult for him to see you. Not a lot of ramp vehicles whizzing around most GA fields.
Most GA fields also close in conditions of poor vis or darkness, unlike the sort of places where HVV do perform a useful function.
Personally I have on in the a/c in case I'm forced to wear it but I would never wear it while in the a/c or while swinging a prop. In those circumstances it's a danger rather than something that mitigates risk. Something that tends to escape the notice of the average jobsworth.