John Carlisle is a consultant anaesthetist at Torbay Hospital on England’s south coast. Unless you’ve been one of his patients, you’ve probably never heard of him. But he’s a researcher too, and he’s developed statistical methods to help spot signs of fraud in medical research.
There’s a public image of medical researchers as being trustworthy people, working hard to make us all healthier. Scientists and doctors come right at the top of an international survey on the trustworthiness of professions. I’d say that image is deserved. But there are cases of very questionable research practice, or even downright fraud.
People usually remember the discredited paper published in The Lancet in 1998, linking the MMR vaccine to autism. Because of the major problems with that research, its lead author, Andrew Wakefield, was struck off the medical register.
The Wakefield case is unusual, though. Most cases of research fraud are harder to spot and aren’t so widely publicised.
Raft of retractions
Carlisle and other anaesthetists got suspicious, more than a decade ago, about studies by a Japanese researcher, Yoshitaka Fujii. He published results from a series of randomised, controlled clinical trials (RCTs) investigating medicines to prevent nausea and vomiting in patients after surgery. Carlisle, and others, thought the data was too tidy to be true. He showed that it was extremely unlikely that some of the patterns in Fujii’s data had occurred by chance. Because of this and further investigation, Fujii lost his university job.
No less than 183 of his papers were retracted, that is, effectively “unpublished” by the journals concerned. That’s far more retractions than any other individual has had.
Since then, Carlisle has developed his methods further. In 2017, he produced an analysis of over 5,000 clinical trials. Most were published in journals in his own field, anaesthetics, but he also included two top-ranking American medical journals, the Journal of the American Medical Association (JAMA) and the New England Journal of Medicine (NEJM). He found suspect data in about 90 papers.
In some cases, there were innocent explanations. But there were several retractions. For instance, a major Spanish trial, investigating whether a Mediterranean diet could help prevent heart diseases and strokes, had to be retracted. The random allocation of people to different diets had, in some cases, been done wrongly. A revised trial report, omitting the wrongly randomised participants, appeared later.
Adopted by medical journals
Carlisle’s methods are now routinely used by at least two medical journals to screen reports of RCTs that are submitted for publication. They are Anaesthesia, in Carlisle’s own specialty, and the prestigious NEJM. Others may well follow.