Point is an employer has a duty of care to the employee.
H&S is about risk, quite agree.
Who judges the risk?
Your employer (or Unit H&S chap/chapess)
You may not like having to wear a hi-vis vest, but it is not your choice.
If the rules require it then wear one, if they do not, then dont.
If you have an accident (or cause an accident) you will not have a leg to stand on (sorry about the pun)