Its the way that people enforce these rules that gets peoples backs up rather than the jackets per se.
Personally I just do not see the justification for them. Shout and dance about seat belts and landing lights, they have a proven benefit. Show me a proven benefit of a high viz jacket.