“Irreproducible results and spurious claims” in neuroscience

The practice of science in today’s “publish or die” world together with the headlong pursuit of funding leaves me somewhat cynical.

My gut feeling has always been that it is the “social sciences” which are plagued most by the “irreproducible study” sickness but it seems to be prevalent across many more disciplines than I would have thought. Poor studies in neuroscience – it would seem – are followed by “meta-studies” to summarise the poor studies and are in turn followed by analysis to prove that the studies are not significant. And poor studies with irreproducible results would seem to be the norm and not the exception.

Gary Stix blogs at The Scientific American:

Brain Lobes : image Scientific American

New Study: Neuroscience Research Gets an “F” for Reliability

Brain studies are  the current darling of the sciences, research capable of garnering  tens or even hundreds of millions in new funding for ambitious new projects, the kind of money that was once reserved only for big physics projects.

Except the house of neuroscience, which attracts tens of thousands of attendees each year to the annual meeting of the Society for Neuroscience, may be built on a foundation of clay. Those are the implications of an analysis published online  April 10 in Nature Reviews Neuroscience, which questions the reliability of much of the research in the field.

The study—led by researchers at the University of Bristol—looked at 48 neuroscience meta-analyses (studies of studies) from 2011 and found that their statistical power reaches only 21 percent, meaning that there is only about a one in five chance that any effect being investigated by the researchers—whether a compound acts as an anti-depressant in rat brains, for instance—will be discovered. Anything that does turn up, moreover, is more likely to be false. …..

John Ioannidis of Stanford University School of Medicine, says ….. “Neuroscience has tremendous potential and it is a very exciting field. However, if it continues to operate with very small studies, its results may not be as credible as one would wish. A combination of small studies with the high popularity of a highly-funded, bandwagon-topic is a high-risk combination and may lead to a lot of irreproducible results and spurious claims for discoveries that are out of proportion.”

Update: Moses Chao, a former president of the Society for Neuroscience and a professor of cell biology at New York University Medical School, got back to me with a comment after I posted the blog, which is excerpted here:

“I agree that many published papers in neuroscience  are based upon small effects or changes.  One issue is that many studies have not been blinded.  There have been numerous reports in my field which have not been reproduced, some dealing with small molecule receptor agonists.  This has set back progress.  The lack of reproducibility is one of the reasons that pharmaceutical companies have reduced their effort in neuroscience research. But irreproducibility also applies to other fields, such as cancer…

Tags: , , ,

%d bloggers like this: