after all Pax supervised by Hi Viz Ramp agents don't have to wear them,
That is because pax are generally walking on prescribed marked walkways.
You know at the end of the day flying is a discipline and some pilots just cannot comply with rules and regulations they do not agree with. If they replaced High Vis jackets with captains jackets and 4 gold stripes I wonder how many would complain then?