Should journals discourage repeat reviews?

I recently reviewed a paper for a journal, and was struck by a check-box in the online reviewer form that I recall encountering before. The check box asked something like, "Would you be willing to have your review forwarded to another journal if the manuscript is submitted elsewhere?" This in turn got me thinking about a broader phenomenon: reviewers reviewing the same manuscript at different journals. I've encountered this phenomenon on social media before, the typical case being a reviewer saying something like, "I rejected this paper at one journal. Then I got asked to review it again at a new journal, and because the author didn't seem to revise it in light of my feedback at the first journal, I just attached my first review advocating rejection at the second journal."

I recognize why some reviewers might want to do these kinds of things. Sometimes you get tasked to review a paper, and it just seems so obviously problematic--and you think you make a good case for that in your review--that you think you would be saving another journal (and referee) time and energy by having yourself review it again. Trust me, I'm sympathetic: I've reviewed papers where this is my own reaction. Despite this, I am extremely uncomfortable with these practices. I am uncomfortable with them, first off all, because I read a lot of history of philosophy and science, and have read case after case of reviewers confidently rejecting papers and books that later went on to enjoy great esteem. Here are just a few:

• Robert Mayer: groundbreaking work on thermodynamics desk-rejected by Annalen der Physik, had to self-publish.
• James Joule: first groundbreaking electrochemistry paper rejected by the Royal Society, and a referee demanded that he remove the entire part of his famous paper establishing conversion of mechanical energy to heat.
• Hermann Helmholtz: groundbreaking paper on conservation of energy rejected by Annalen der Physik.
• Wittgenstein's Tractatus: rejected by something like a dozen different presses.

There are also more contemporary cases ready to hand. For instance, a while back Jason Stanley shared how some of his most influential papers were systematically rejected by journals--and I've heard similar stories from others.

These sorts of stories should, I think, lead reviewers--and, I would suggest, editorial policies--to err on the side of epistemic humility. Yes, you may be convinced that the paper you just reviewed is the Worst, Most Unpublishable Paper ever...yet this is apparently how some of the most important papers in history were viewed by reviewers. In every case where you are confident a paper is unpublishable, it might seem to you unfathomable that you could be wrong about it. Still, given that there have been numerous reviewers in history in just your position who not only got it wrong but indeed spectacularly wrong, I think there is at least one reason to adopt the epistemically humble position, erring on the side of letting new referees review the paper if it is submitted at a new journal.

Further, I think there are other reasons to err in this direction. One of the things I have noticed reading the history of this sort of thing is that it often appears to be unacknowledged biases that lead experts to get things so wrong. For instance, although his papers were published, many eminent physicists strongly denounced Einstein's relativity papers, even after there was observational evidence of bending starlight! For example, then-president of the British Royal Society and 1906 Nobel Prize winner in physics dismissed Einstein, saying, "no one has yet succeeded in stating in clear language what the theory of Einstein really is." Similarly, Sir Oliver Lodge, chair of the University of Chicago's physics department, dismissed the theory as "repugnant to commonsense." (Einstein: A Life, pp. 101-2). And head of the astronomy department at Chicago? He questioned Einstein's credentials as a patent clerk adding, "The Einstein theory is a fallacy. The theory that the 'ether' does not exist...is a disgrace to our age." (Ibid, pp. 102-3) Then there is Frege's reaction to Wittgenstein's Tractatus, which I recently recounted here--where Frege essentially dismissed Wittgenstein's now-famous work because it didn't conform to standard conventions ("One expected to see a question, to have a problem outlined, to which the book would address itself. Instead, one came across a bald assertion, without being given the grounds for it..."). Whatever one thinks of Frege's view here (I know there are some that concur with him, though I don't!), the point is that all of these cases--including the cases of Mayer, Joule, Helmholtz, and many others--show that, as certain as one may be that a work is poor or misguided, sometimes our judgment can be corrupted by biases, ones that other readers/reviewers may not share. This, in my view, is a second reason to err on the side of epistemic humility--a second reason not to review a paper again for another journal or have one's review forwarded.

Finally, I think there is a third reason for the same attitude: the moral risks involved. Allow me to explain. First, it is no secret that we work in a "publish or perish" discipline (and indeed, some people publish and still perish!). Researchers need publications to receive jobs, obtain tenure, etc. Second, the peer-review process is already excessively long. While some journals have two or three month turnaround times, other journals can take many months or well over a year. Consequently, I think there is some real moral risk--in terms of harming authors' career prospects--to the mere possibility of both (A) getting things wrong with a paper (viz. advocating rejection as a reviewer), and (B) holding a particular paper to that judgment at multiple journals. For, quite realistically, if both (A) and (B) are the case--if one not only gets things wrong as a review and prevents the paper's publication at multiple journals--you have, in effect, outsized negative effects on the author's research output. Given how little one has invested in the matter as a reviewer (the time one spends reviewing has little bearing on one's career prospects), compared to just how much a researcher's career can depend on their publishing record, this seems to me yet another reason to err on the side of epistemic humility: yet another reason to give the author a chance with new reviewers.

Is it possible that allowing a paper to be reviewed by others may be a waste of their time? Sure it is. However, if a paper is really as bad as you think it is, then it really shouldn't be that hard for editors and other reviewers to spot the problems as well, in which case it won't be much skin off their back to reject the paper themselves. All in all, then--counting up the reasons for and against reviewing a paper multiple times for multiple journals--the balance of reasons seems to me firmly against the practice. Personally, I am inclined to think that journals should not only not give authors the option of forwarding their review to other journals, but that editors should also ask reviewers to disclose and recuse themselves if they have reviewed a given paper at a previous journal. Why? Again, because of the serious risks of (wrongly) harming someone's career through potential biases--risks that can be avoided by simply giving the author a chance with new reviewers (something which again, in my view, carries few if any serious counter-costs, since it is reasonable to assume that a truly bad paper will be judged as such by other reviewers).

But these are just my thoughts. What are yours?

Comments

[Reviewer]

I do not agree to review a paper that I have reviewed before, for another journal. This is a firm policy with me. I think the person deserves another chance. But there are cases where I have been asked to review the same paper three and even four times by different journals. I did it the first time, and I always say I have reviewed the paper before. One tricky author even changed the title a few times ... and it kept getting sent to me. I think I have a reputation for fast turn around times on reviews (I average about 2 days). I only agree to review when I know I can get it done in the next day or two.

[Marcus Arvan]

Hi Reviewer: Obviously, I don't anything about your performance as a reviewer in other respects--but in these respects you seem to be an ideal reviewer. Speaking as an author, we should all be so lucky!

[Marcus Arvan]

This article "How Are the Mighty Fallen: Rejected Classic Articles by Leading Economists" (posted by Ingrid Robeyns on another thread) is particularly eye-opening and relevant: https://www.aeaweb.org/articles?id=10.1257/jep.8.1.165

There are a ton of gems in the paper. Everyone should read it from beginning to end. This passage--about some of the harms that can be imposed on authors--is particularly dismaying:

"However, like Becker, Krugman notes that, even if another journal eventually prints a paper, the delay that initial rejection causes may permit others to beat the paper into the intellectual market. Krugman sent his "Target Zones and Exchange Rate Dynamics" to the Journal of Political Economy. "This time I got two favorable referee reports. The paper was nonetheless rejected ... by [the referee] who thought that the paper was of 'insufficient general interest' for the JPE. The paper didn't come out (in the QJE) until August 1991. By that time the target zone literature, all of which made use of the techniques first introduced in my paper, had exploded, and consisted of at least a hundred published and unpublished pieces; in fact, I had to add a postscript to the QJE version referring to subsequent literature."

[Peter]

I tend to agree with your main claim. I know that there is already strain on the system, but I think that this is not the place to try to relieve some of this strain. At the very least, I think referees should disclose the fact that they refereed this paper before. In general, though, I think they should just reject the invitation to review.

[NS]

I think Doron Zeilberger has an interesting take on the matter driving the moral risk. http://sites.math.rutgers.edu/~zeilberg/Opinion107.html Not reusing reviews seems to just be stepping away from the central problem of referees being too harsh.

[Steven French]

I'm not convinced of the relevance of the examples from the history of science but thats another issue ... I also feel uncomfortable refereeing papers that Ive seen before, for broadly the reasons outlined here. And I'm heartened by the number of referees who tell us they've already seen a particular paper, either giving us the option of asking someone else or explicitly declining.

[Tim]

Since papers are often rejected outright even if one reviewer recommended acceptance or revisions, it should depend on what you as the reviewer recommended originally. Unless your recommendation was rejection, I see no principled reason never to review the paper again. I don't have a policy on what to do when my original recommendation was to reject the paper (happened only once so far), but I am wondering how to implement the proposal: Marcus, is your idea to decline without giving a reason for doing so or to decline while telling the journal that you have reviewed the paper before? Not telling the journal why you decline is awkward (you don't want to come across as a bad colleague), telling them may harm the paper more than actually sending your full report...

[Marcus Arvan]

Hi Tim: That's a very good point! I guess I'm inclined to say that if you recommended acceptance or an R&R, then one *should* agree to review again--simply because the moral risks in that case are far less than in the case of advocating rejection (viz. holding a paper's publication prospects "hostage" to your negative review).

Consequently, I think I'm willing to amend my proposal to hold that one should not agree to review a paper again only in the case where one advocates rejection.

In any case, in these cases I think of course it makes sense to let the journal know. I think I would simply say something like, "I decline to review the paper because I reviewed it before and out of intellectual humility, I think the paper should have a shot with new reviewers." I don't see how telling a journal this should harm the paper--at least not if an editor is doing their job properly (in this case, I think the *editor* should express the same kind of intellectual humility as the reviewer, giving the paper a shot with new reviewers).

[Tim]

As a rule of thumb: If you hint at something negative, it should be as specific as possible. Thus, we shouldn't say "I think he isn't an entirely honest person" (but I'm not telling you why I'm suspicious), but rather "I think he isn't an entirely honest person because it seems that he lied about his GPA". Since the latter is specific, others can decide for themselves whether he actually lied and whether lying about one's GPA makes one a dishonest person. The former, however, will plant a lingering doubt. Analogously: "There might be something seriously wrong with this paper, but please ask someone else" vs. "this paper may rest on a misinterpretation of Davidson, but please ask someone else". For a journal editor the latter should be more useful than the former. They just need to ask a Davidson expert whether the interpretation is defensible. – Maybe there are two conceptions of epistemic humility in play here. Not saying anything ("I just can't tell whether he is a honest person") vs. saying something while hedging it and being open-minded ("I think he's a dishonest person because XYZ, but please correct my impression if yours is different").

[Pendaran]

I was asked before to review a paper I had previously rejected at another journal. I declined saying the author should have the chance at a different opinion. This is called humility. I didn't think the paper was good enough, but I am just one person. Every philosopher knows how much disagreement there is in the profession. And even if your criticisms are accurate that doesn't mean that someone else might not rightly see something worthwhile in the paper. Perhaps someone else would see how it can be revised to be publishable. We also all have our own biases, as has been mentioned before, and these biases may hinder our ability to assess a paper from a more objective standpoint.