Replication studies
Should nothing new, nothing novel be the reason to reject a manuscript?
New England Journal of Medicine, Journal of American Medical Association, Lancet and High-impact-factor (>7) medical specialty journals from 1990 to 2003 were scanned, from which papers that were cited more than 1000 times were selected. Altogether 49 highly cited original clinical studies were compared against subsequent studies of comparable or larger sample size and similar or better controlled designs. Approximately, 1/3 of those studies were later found to be incorrect (Ioannidis’05).
Let that sink in a bit… those 49 papers are representative of the best papers published in the highly prestigious, highly selective journals. If 1/3 of the best of the best papers got it wrong, what would be the chances of the papers by mere mortals getting it right?
At one point, AMGEN, a pharmaceutical company, was heralded as the most innovative company in the US. Their R&D budget/income was looked at. They heavily invest in research looking for new chemicals to make a new drug. This is for profit entity. It’s always been on the look out for a new drug. They sift through publications looking for potential chemicals that can be turned into a mega hit drugs. Amgen tried to replicate 53 ‘land mark’ (i.e. classic, highly cited) oncological studies, often co-operating closely with the original researchers to ensure that their experimental technique matched the one used first time round. 89% cannot be replicated (Begley & Ellis’12).
The idea that the same experiments always get the same results, no matter who performs them, is one of the cornerstones of science’s claim to objective truth. If a systematic campaign of replication does not lead to the same results, then either the original research is flawed (as the replicators claim) or the replications are (as many of the original researchers like to believe). Either way, something is awry (The Economist’13).
Hardly, if ever, one study would change the field, the idea or the way we practice. It takes multiple studies conducted by different research groups, in different locations, at different points in time to corroborate the findings. It IS about body of evidence; evidence as a whole and not one single study that pushes us forward. If replication studies get rejected, we wouldn’t have studies to confirm nor contradict the first study and we wouldn’t be getting closer to the truth. The first, the new and the novel might’ve got it wrong.
Replication studies may neither be sexy, exciting, nor ground-breaking, but they are performing a significant function in the pursuit of the truth, they are important nonetheless. They should not be rejected just because they are not new nor novel.
Changing the attitude of scientists, granting bodies, reviewers and journal editors to see the merits of replication studies is no less a Herculean task, but in the spirit of truth seeking, we all must.
Sources:
Trouble at the lab. The Economist. 2013 Oct
Ioannidis JP. Contradicting and initially stronger effects in highly cited clinical research. JAMA. 2005 Jul 13;294(2):218-28.
Begley CG, Ellis LM. Drug development: Raise standards for preclinical cancer research. Nature. 2012 Mar 28;483(7391):531-3