Attending to someone who has been hit down without wearing a Hi Viz tabard on an airfield certainly changed my view many years ago on wearing hi viz. Although a degree of discression should be allowed such as when a group of crew walk across the apron with Hi Viz tabards on & one member in the middle has not got one on, as long as they stay close together, they will probably get away with it on this occaison, after all Pax supervised by Hi Viz Ramp agents don't have to wear them, crew are required to especially doing their walk around checks with equipment in use servicing their aircraft.
I frequently hear crew say "OK what have we done wrong now?" as I walk up the steps to speak to the them.
I agree there are some heavy handed Ramp safety teams around who tend to be over the top.