Researchers show that peer review is easily corrupted

PhysicsWorld reports on a new paper:
Peer-review in a world with rational scientists: Toward selection of the average
by Stefan Thurner and Rudolf Hanel
1Section of Science of Complex Systems, Medical University of Vienna, Spitalgasse 23, A-1090, Austria

Just a small number of bad referees can significantly undermine the ability of the peer-review system to select the best scientific papers… Scholarly peer review is the commonly accepted procedure for assessing the quality of research before it is published in academic journals. It relies on a community of experts within a narrow field of expertise to have both the knowledge and the time to provide comprehensive reviews of academic manuscripts.Stefan Thurner and Rudolf Hanel at the Medical University of Vienna created a model of a generic specialist field where referees, selected at random, can fall into one of five categories. There are the “correct” who accept the good papers and reject the bad. There are the “altruists” and the “misanthropists”, who accept or reject all papers respectively. Then there are the “rational”, who reject papers that might draw attention away from their own work. And finally, there are the “random” who are not qualified to judge the quality of a paper because of incompetence or lack of time.Within this model community, the quality of scientists is assumed to follow a Gaussian distribution where each scientist produces one new paper every two time-units, the quality reflecting an author’s ability. At every step in the model, each new paper is passed to two referees chosen at random from the community, with self-review excluded, with a reviewer being allowed to either accept or reject the paper. The paper is published if both reviewers approve the paper, and rejected if they both do not like it. If the reviewers are divided, the paper gets accepted with a probability of 0.5.

Peer review gauntlet

Thurner and Hanel find that even a small presence of rational or random referees can significantly reduce the quality of published papers. Daniel Kennefick, a cosmologist at the University of Arkansas with a special interest in sociology, believes that the study exposes the vulnerability of peer review when referees are not accountable for their decisions.

Kennefick feels that the current system also encourages scientists to publish findings that may not offer much of an advance. “Many authors are nowadays determined to achieve publication for publication’s sake, in an effort to secure an academic position and are not particularly swayed by the argument that it is in their own interests not to publish an incorrect article.”

(This could have been written about Marc Hauser —

But Tim Smith, senior publisher for New Journal of Physics feels that the study overlooks the role of journal editors. “Peer-review is certainly not flawless and alternatives to the current process will continue to be proposed. In relation to this study however, one shouldn’t ignore the role played by journal editors and Boards in accounting for potential conflicts of interest, and preserving the integrity of the referee selection and decision-making processes,” he says.

In fact Journal Editors have much to answer for in the perversion of the peer review process which was revealed by Climategate. (The Hockey Stick Illusion by Andrew Montfordreviewed here)

Thurner argues that science would benefit from the creation of a “market for scientific work”. He envisages a situation where journal editors and their “scouts” search preprint servers for the most innovative papers before approaching authors with an offer of publication. The best papers, he believes, would naturally be picked up by a number of editors leaving it up to authors to choose their journal. “Papers that no-one wants to publish remain on the server and are open to everyone – but without the ‘prestigious’ quality stamp of a journal,” Thurner explains.

When reviewers show bias (in acceptance or in rejection) or misuse and hide behind the cloak of anonymity and are not required to be accountable then Hausergate and Climategate become inevitable.

Oliver Manuel comments: The most basic problem with ANONYMOUS peer-review is this: “That methodology is flawed and those flaws have been gradually undermining, corrupting, and trivializing American science for decades.” Anonymous peer review of papers and proposals has been steadily “undermining, corrupting, and trivializing American science” since I started my research career in 1960.

The evolution of peer review with the use of open servers in now overdue but is beginning.

Tags: , , , , ,

%d bloggers like this: