It is not suggested that the bias is any form of deliberate misconduct but a new paper shows that animal studies are subject to an “excess significance bias”.
Tsilidis KK, Panagiotou OA, Sena ES, Aretouli E, Evangelou E, et al. (2013) Evaluation of Excess Significance Bias in Animal Studies of Neurological Diseases. PLoS Biol 11(7): e1001609. doi:10.1371/journal.pbio.1001609
Author Summary
Studies have shown that the results of animal biomedical experiments fail to translate into human clinical trials; this could be attributed either to real differences in the underlying biology between humans and animals, to shortcomings in the experimental design, or to bias in the reporting of results from the animal studies. We use a statistical technique to evaluate whether the number of published animal studies with “positive” (statistically significant) results is too large to be true. We assess 4,445 animal studies for 160 candidate treatments of neurological disorders, and observe that 1,719 of them have a “positive” result, whereas only 919 studies would a priori be expected to have such a result. According to our methodology, only eight of the 160 evaluated treatments should have been subsequently tested in humans. In summary, we judge that there are too many animal studies with “positive” results in the neurological disorder literature, and we discuss the reasons and potential remedies for this phenomenon.
Roli Roberts writes at the PLOS blog:
But a study just published in PLOS Biology by Konstantinos Tsilidis, John Ioannidis and colleagues at Stanford University shows that a meta-analysis is only as good as the scientific literature that it uses. That literature seems to be compromised by substantial bias in the reporting of animal studies and may be giving us a misleading picture of the chances that potential treatments will work in humans. ….
Rather than wilful fraud, the authors of the PLOS Biology study suggest that this excess significance comes from two main sources. The first is that scientists conducting an animal study might analyse their data in several different ways, but ultimately tend to pick the method that gives them the “better” result. The second arises because scientists usually want to publish in higher profile journals that tend to strongly prefer studies with positive, rather than negative, results. This can delay or even prevent publication, or relegate the study to a low-visibility journal, all of which reduce their chances of inclusion in a meta-analysis.
The new work raises important questions about the way in which the scientific literature works, and it’s possible that the types of bias reported in the PLOS Biology paper have been responsible for the inappropriate movement of treatments from animal studies into human clinical trials. What do we do about it? Here are the authors’ suggestions:
- Animal studies should adhere to strict guidelines (such as the ARRIVE guidelines) as to study design and analysis.
- Availability of methodological details and raw data would make it easier for other scientists to verify published studies.
- Animal studies (like human clinical trials) should be pre-registered so that publication of the outcome, however negative, is ensured.
Well, these are all excellent, but most people would also say that there are problems elsewhere in the system – in the high-profile journals’ desire to a have a cute story with well-defined conclusions, and in the forces exerted on authors by institutions and funding bodies to publish in those high-profile journals.
The k2p blog 