Originally Posted by
Richard Dangle
PS
Just to be clear, I'm not dissing the study in anyway whatsoever. I'm saying if you want to understand it, you need to read it.
There is a curious passage in the "Limitations" section. My main thought as I was reading it was that they were doing a
lot of independent tests and that the p-value correction (for multiple comparisons) must be eating up a lot of their power. It turns out that they hadn't adjusted the tests at all. They write: "
Had we adjusted the alpha level for multiple comparisons, we may have obtained fewer “statistically significant” results but at the expense of increasing the false negative (or missed discovery) risk. In our judgment, false positives would be less concerning than false negatives."
No real criticism of the authors as they are open about the methods, but if your aim is to avoid false negatives, then why not just choose a higher threshold for the family-wise error rate? The results would at least be a lot more intuitive to the reader. I suspect that this approach wouldn't pass review if submitted to a journal, though possibly the purpose of this report was slightly different to an academic study.
I also wonder whether a false positive would really be less concerning if it were being described to a former fighter pilot!