Spectacular failures to replicate key scientific findings have been documented of late, particularly in biology, psychology and medicine.
A report on the issue, published in Nature this May, found that about 90% of some 1,576 researchers surveyed now believe there is a reproducibility crisis in science.
While this rightly tarnishes the public belief in science, it also has serious consequences for governments and philanthropic agencies that fund research, as well as the pharmaceutical and biotechnology sectors. It means they could be wasting billions of dollars on research each year.
One contributing factor is easily identified. It is the high rate of so-called false discoveries in the literature. They are false-positive findings and lead to the erroneous perception that a definitive scientific discovery has been made.
This high rate occurs because the studies that are published often have low statistical power to identify a genuine discovery when it is there, and the effects being sought are often small.
Further, dubious scientific practices boost the chance of finding a statistically significant result, usually at a probability of less than one in 20. In fact, our probability threshold for acceptance of a discovery should be more stringent, just as it is for discoveries of new particles in physics.
The English mathematician and the father of computing Charles Babbage noted the problem in his 1830 book Reflections on the Decline of Science in England, and on Some of Its Causes. He formally split these practices into “hoaxing, forging, trimming and cooking”.
‘Trimming and cooking’ the data today
In the current jargon, trimming and cooking include failing to report all the data, all the experimental conditions, all the statistics and reworking the probabilities until they appear significant.
The frequency of many of these indefensible practices is above 50%, as reported by scientists themselves when they are given some incentive for telling the truth.
The English philosopher Francis Bacon wrote almost 400 years ago that we are influenced more by affirmation than negatives and added:
Man prefers to believe what he prefers to be true.
Deep-seated cognitive biases, consciously and unconsciously, drive scientific corner-cutting in the name of discovery.
This includes fiddling the primary hypothesis being tested after knowing the actual results or fiddling the statistical tests, the data or both until a statistically significant result is found. Such practices are common.
Even large randomised controlled clinical trials published in the leading medical journals are affected (see compare-trials.org) – despite research plans being specified and registered before the trial starts.
Researchers rarely stick exactly to the plans (about 15% do). Instead, they commonly remove registered planned outcomes (which are presumably negative) and add unregistered ones (which are presumably positive).