How anonymous peer review fails to do its job and damages science.

by JP de Ruiter, Bielefeld University

Churchill believed that democracy was the “worst form of government except all those other forms that have been tried from time to time.” Something analogous is often said about anonymous peer review (APR) in science: “it may have its flaws, but it’s the ‘least bad’ of all possible systems.” In this contribution, I present some arguments to the contrary. I believe that APR is threatening scientific progress, and therefore that it urgently needs to be fixed.

The reason we have a review system in the first place is to uphold basic standards of scientific quality. The two main goals of a review system are to minimize both the number of bad studies that are accepted for publication and the number of good studies that are rejected for publication. Borrowing terminology of signal detection theory, let’s call these false positives and false negatives respectively.

It is often implicitly assumed that minimizing the number of false positives is the primary goal of APR. However, signal detection theory tells us that reducing the number of false positives inevitably leads to an increase in the rate of false negatives. I want to draw attention here to the fact that the cost of false negatives is both invisible and potentially very high. It is invisible, obviously, because we never get to see the good work that was rejected for the wrong reasons. And the cost is high, because it removes not only good papers from our scientific discourse, but also entire scientists. I personally know a number of very talented and promising young scientists who first sent their work to a journal, fully expecting to be scrutinized, but then receiving reviews that were so personal, rude, scathing, and above all, unfair, that they decided to look for another profession and never looked back. I also know a large number of talented young scientists who are still in the game, but who suffer intensely every time they attempt to publish something and get trashed by anonymous reviewers. I would not be surprised if they also leave academia soon. The inherent conservatism in APR means that people with new, original approaches to old problems run the risk of being shut out, humiliated, and consequently chased away from academia. In the short term, this is to the advantage of the established scientists who do not like their work to be challenged. In the long run, this is obviously very damaging for science. This is especially true of the many journals that will only accept papers that receive unanimously positive reviews. These journals are not facilitating scientific progress, because work with even the faintest hint of controversy is almost automatically rejected.

With all this in mind, it is somewhat surprising that APR also fails to keep out many obviously bad papers.

There are many problems with APR, but I suspect the two biggest reasons for its failure are:

• Reviewers have little time (note that they are not paid for their work) leading to sloppy and superficial reviews.
• Reviewers often do not understand the difference between studies with results that they don’t like and studies that have scientific flaws. This effectively leads to reviewers using their power to eliminate or hamper competitors.

It is important to realize that if we look at peer review as a strategic game, rejecting everything is a strong strategy, as this will always reduce the influence of the (reviewer’s) competition. This strategy, if generally applied, would be catastrophic for scientific progress. Indeed, I know of at least one scientific field where grant reviewers nearly always mutually reject the proposals of their peers, to the bemusement and advantage of other, more cooperative fields, who then have more funding available.

I believe that the most fundamental problem in APR is accountability. Reviewers can basically say whatever they want, because they are protected by anonymity. To illustrate, here are a few comments (used to justify outright rejections) that I have received over the years. Note that I’m not saying that these papers didn’t deserve to be rejected; only that these specific arguments to reject them are not valid and/or not fair.

• These empirical results are wrong, because they contradict influential theory X.
• Those experimental subjects must have been doing it on purpose.
• This work is only interesting for cocktail parties at the Max Planck Institute.
• The phenomenon that you have studied actually doesn’t exist.
• I cannot see any flaws in these experiments, and the author has addressed all my concerns, but I have to say I still just don’t like this study.

I suspect we all have a personal top ten list of these types of reviewer statements. But would these reviewers have written those things if they had been identified by name, and if their reviews had been accessible to the scientific community at large? Do we find arguments of this sort in open peer reviews, like for instance the commentaries in Behavioral & Brain Sciences? Of course not. It is anonymity that enables reviewers to write comments like that. It is dysfunctional APR culture that allows reviewers to then persuade editors (who, let’s not kid ourselves, don’t have the time to read and check everything in detail) to reject papers on such weak grounds. If these types of argument were used to reject the first paper by a young scientist who submitted a dissertation project, for example, it would be very demotivating and unfair.

The party line on why we need anonymity for reviewers is exemplified by the scenario of a young, untenured scientist being asked to review the work of a famous, politically powerful peer. If these young scientists were not anonymous, so the argument goes, they could not freely criticize the work of more famous scientists without fearing consequences for their career. Good point. Note, however, that if we assume that those famous scientists are potentially vengeful towards a young reviewer who “wronged” them, we also have to assume that they would be equally unpleasant when reviewing work by younger scientists that criticized their own. So we employ APR to protect the young scientists from the wrath of their more famous colleagues, but by the same token we allow the famous scientists to block the work of the younger colleagues whenever they find it unpalatable, and, moreover, to do so anonymously. This is a clear case of well-intentioned social engineering backfiring in a disastrous way.

There are ways of improving this situation considerably. I am specifically thinking of two rules. The first one I have implemented for myself personally, and I urge others to do so as well. The second one I’ve tried to sell to journal editors, but with very little success so far.

Rule a) Reviewers with tenure always sign their reviews.

Rule b) Reviews are stored, and all researchers have the explicit right to look up and cite reviews. If the author of a certain review is anonymous, so be it. Call them “reviewer 3 in submission so-and-so to journal X”, but at least this allows researchers to address and discuss their arguments in the papers. I often notice that reviewers have a very strong influence on papers, by requesting that certain points be addressed before they advise acceptance. This epistemic tug-of-war between reviewers and authors often results in needless meandering and bad rhetorical flow.

Under the current system, the very arguments used by gatekeepers to decide whether a paper lives or dies, and the influence reviewers have on the paper’s content in case it lives, are unaccountable, unknown, and therefore not open to scrutiny. Having reviews stored, accessible, and addressable would make gatekeepers officially part of the discourse. This would reduce both the false positives and the false negatives, because reviewers who reject good papers and reviewers who accept bad papers, for whatever reason, can be held accountable for what they write. Also, on a more positive note, reviewers would get more credit for their work. Under the current system, the difference between being a constructive reviewer and a careless one is invisible to all except journal editors.

Having a review system is probably necessary to keep our standards high. What we need to change is reviewers’ accountability for what they write; we need their arguments to become an integral part of the scientific discourse. As long as reviewers are shielded by anonymity, and their arguments hidden in editors’ file drawers, the review system is highly vulnerable to incompetence, abuse of power, and corruption.