Latest accident figures I can find for the USA, where what you wear only really depends on the weather, are from
2001 (PDF)
From what I can tell, only four people on the ground were killed by collisions with a plane, and only one by contact with a prop/rotor, and that was a security guard who walked
into a rotor. One other person was injured. Two were killed in a car hit by a landing overun and would probably not have been saved by wearing a vest. So it would appear that this rule has no demonstrable basis in safety, if it's a safety argument then we should all wear a hi-vis vest when out shopping.