The only way to run these is to preregister the the criteria end points prior to running the experiment. There is a large push for studies to do exactly this in modern science. A significant proportion of medical & psychological research has been found to be faulty even though poportially i gives the external appearance of going through a rigorous scientific process. It is all too easy to come to a conclusion after collecting and analysing the data.
The preregistration revolution
Abstract
Progress in science relies in part on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated conceptually but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings. However, ordinary biases in human reasoning, such as hindsight bias, make it hard to avoid this mistake. An effective solution is to define the research questions and analysis plan before observing the research outcomes—a process called preregistration. Preregistration distinguishes analyses and outcomes that result from predictions from those that result from postdictions. A variety of practical strategies are available to make the best possible use of preregistration in circumstances that fall short of the ideal application, such as when the data are preexisting. Services are now available for preregistration across all disciplines, facilitating a rapid increase in the practice. Widespread adoption of preregistration will increase distinctiveness between hypothesis generation and hypothesis testing and will improve the credibility of research findings.
One psychologist has even gone to the extent of "proving" paranormal psychology using unimpeachable and rigorous science to make the point using standard scientific methods. He has passed every conventional scientific hurdle:
THE CONTROL GROUP IS OUT OF CONTROL
Bem, Tressoldi, Rabeyron, and Duggan (2014), full text available for download at the top bar of the link above, is parapsychology’s way of saying “thanks but no thanks” to the idea of a more rigorous scientific paradigm making them quietly wither away.
You might remember Bem as the prestigious establishment psychologist who decided to try his hand at parapsychology and to his and everyone else’s surprise got positive results. Everyone had a lot of criticisms, some of which were
very very good, and the study
failed replication several times. Case closed, right?
Earlier this month Bem came back with a meta-analysis of ninety replications from tens of thousands of participants in thirty three laboratories in fourteen countries confirming his original finding, p < 1.2 * -1010, Bayes factor 7.4 * 109, funnel plot beautifully symmetrical, p-hacking curve nice and right-skewed, Orwin fail-safe n of 559, et cetera, et cetera, et cetera.
The bottom line in all this is science in the service of business, usually Medical & pharmaceutical, but not limited to these has let itself become a tool of generating desired outcomes. Science can be used to obtain the truth and for further human knowledge and progress. Unfortunately it can also be used for nefarious ends and become corrupt beyond belief. Many fancy and impressive reports and studies are just straight out wrong. Editors of both
the Lancet and the
New England Journal of Medicine have come out recently and said the following
Originally Posted by The Lancet
The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue.
Originally Posted by Marcia Angell NEMJ Editor
Journal editors have expended much time and effort in teasing out how to handle authors' and reviewers' competing interests. They need now to concentrate on their own and those of their employers, lest we reach the dismal scenario described by Marcia Angell: “it is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of The New England Journal of Medicine
If you have a spare few hours, listen to
Peter Attia interview with Katherine Eban on the widespread fraud in the Generic drug sector. Almost certainly all of us who consumed a potentially harmful non compliant medication if we have used a generic drug over the last 20 years. The generic drug company were masters at fooling the FDA with straight out data fraud. In the case of Ranbaxy, over 200 medications approved by the FDA entered the global supply chain based on totally bogus and data. This is a very disturbing story about just how corrupt science in the service of profit has become.
So no, I would not be putting much faith in the outcome of a minuscule sample size when the company with a vested interest in the outcome is paying the bill. If the study was preregistered and followed a larger cohort over a significant period of time with open raw data available I would have faith in the outcome.