Hi-viz might be seen as a contributory factor in the deliberations, but for example if you walk into a turning propeller it's likely to be your fault whatever you're wearing; conversely if a pilot taxis with instrument flying screens up and runs someone over it's likely to be the pilot's fault whatever the victim was wearing.
Even the HSE are trying to de-bunk these Health and Safety myths.
True - and if you had a hi-viz jacket on and had a stroke, it's not your fault either, and will not save your life. But it just keeps you safe by making you far more visible in an environment with rotating props and fast moving lumps of metal. I don't see how that can be a bad thing. As long as you take them off before getting into the aircraft, hi viz are brilliant. We are criticised for not wearing them at my airfield and rightly so.