Creampuff (bless his socks) wrote:
My favourite paper is David H. Freedman's Wrong: Why Experts Keep Failing Us—And How to Know When Not to Trust Them. My favourite quotes from that paper:
“Researchers headed into their studies wanting certain results—and, lo and behold, they were getting them.”
“At every step in the process, there is room to distort results, a way to make a stronger claim or to select what is going to be concluded,”
“Not surprisingly, the studies that tend to make the grade are those with eye-catching findings. But while coming up with eye-catching theories is relatively easy, getting reality to bear them out is another matter. The great majority collapse under the weight of contradictory data when studied rigorously.”
“[R]esearchers were frequently manipulating data analyses, chasing career-advancing findings rather than good science ...”
“[A]ssuming modest levels of researcher bias, typically imperfect research techniques, and the well-known tendency to focus on exciting rather than highly plausible theories, researchers will come up with wrong findings most of the time.”
“Simply put, if you’re attracted to ideas that have a good chance of being wrong, and if you’re motivated to prove them right, and if you have a little wiggle room in how you assemble the evidence, you’ll probably succeed in proving wrong theories right.”
The Pape and Denison hearings of the ‘80’s devoted a lot of energy to examining the “scientific” evidence of the day, the bulk of which came from the Vic College of Optometry. It is amazing how the same names just keep on popping up 25 years later. In those AAT hearings we mounted a sustained critique of the works written by Cole, Vingrys, MacDonald, and Bowman, to name just a few. The six paragraphs in the above quote were applicable to these works. None of the works presented by the then DOT were able to withstand the critiques offered, and the cases were lost on the basis of the bad evidence that the regulator presented through their proxies, the optometrists. There are more modern additions to the list, and the problem their works suffer from come from the same five points in the above quote.
It should be noted that these authors, both the old and the new, come from an industry sector that earns a great deal of income from the promotion of, and implementation of industrial colour perception standards. There is a multi-million dollar industry at stake. It didn’t shock me when I attended my first ASMA (Aerospace Medical Association Meeting) in Chicago last year and found a multitude of new colour vision tests on display, all vying for a piece of the action in the resurgence of aggressive promotion of the aviation colour perception standard. The marketing of the CAD test has been brilliant, as far as marketing goes. I wonder who started the CAD project off, the CAA UK or the City University, London, Applied Vision Research Department. Irrespective of the answer, the marketing claim that the CAD test is “aviation specific” is demonstratively untrue and not supported by the documentation provided. Make no mistake, the CAD alone represents a very large investment in the expectation of equally large profits. But, in the debate that is now raging in good old Oz, the CAD and its underlying philosophy, will be given the critical scrutiny it deserves.
I am so glad to see that so many who have contributed to this thread “get it” that the only tests that have validity in assessing the safe performance of pilots are those tests that measure the safe performance of pilots, i.e. the flight tests, simulator sessions, check and training tests, instrument renewals and so forth. They “get it” that in the entire history of aviation there has been only one accident in which the colour vision deficient status of a pilot was attributed a causal role, and that there are more plausible explanations for that crash, which if accepted, leave the tally of accidents due to CVD at NIL. Not bad for a century old industry.