Leading medical journal The Lancet has launched an attack on what it calls the systematic review ‘gravy train’ saying it is a fundamentally flawed process due to the inclusion of small, biased trials.
In a scathing commentary article, the Lancet’s editor Dr Richard Horton takes aim at the Cochrane Library and funding bodies such as Australia’s NHMRC for supporting the growth of a mammoth industry of systematic reviews and meta-analyses that may be ‘poisoning’ medical practice rather than nurturing it.
The Lancet editorial says it is no longer possible to ignore the strong doubts about the scientific validity of systematic reviews raised by leaders of evidence-based medicine such as Ian Roberts, Professor of Epidemiology and Public Health and Co-Director of the Clinical Trials Unit at the London School of Hygiene & Tropical Medicine.
Professor Roberts and others have described the widespread belief and support for systematic reviews in healthcare as an “anachronistic religion” because reviews often include low quality trials with a strong risk of bias.
One of his strongest criticisms is that systematic reviewers ‘sweep up’ small and underpowered trials into their meta-analyses without applying sufficiently rigorous inclusion criteria.
Systematic reviews may also be giving the wrong answer due to selection bias and because their findings are strongly influenced on the inclusion of one or two large but false ‘randomised trials’ , it adds.
This highlights the other fatal flaw of systematic reviews – that they are based on trust in the integrity of trial design and results.
“Trials are … taken at face value without fact-checking. And they are likely to include false-positive results, rendering the final review not only unreliable but also quite possibly wrong,” the editorial states.
Trust could be restored if reviewers included only prospectively registered trials and checked that trial data were real and accurate, the editorial suggests.
Dr Horton says the valid criticisms are not being taken seriously by the ‘powerful organisations’ that have emerged to construct a knowledge base in medicine underpinned by the results of systematic reviews.
“[The] NIHR, NIH, and NHMRC have good reasons to silence a debate about the value of their massive investments into the systematic review industry,” he writes.
He concludes by echoing the sentiments of Professor Roberts that it is time for those responsible for producing clinical guidelines that inform decision making to listen to the critics.