When editors ignore your reviews?

In our newest "how can we help you?" thread, Frustrated writes:

I have a question about good practices for reviewers. In the past few months this happened to me 3 times. I was asked to review a paper for a top journal in my AOS. This paper is really really bad. It's bad in the following sense: the topic is important, the thesis is interesting, but it'so poorly written and argued that, if I did something like that myself, I'd get for sure a rejection. However, I recommend major revisions. After a few weeks I got the revisions back, and nothing much has changed. I can see what the other reviewer has suggested, and it's minor revision. I then suggest rejection and the paper is accepted. From this I understand that the paper was either from an established scholar, or from a student of an established scholar. I feel I'm wasting my time by doing my diligent work of reviewer, given that the editor has already decided. Therefore, this is my question: when I have the feeling that something similar is happening again, shall I address the editor directly and ask how I should treat the paper? Or shall I just decline to review the revisions?

I'm glad Frustrated wrote in about this, as I think it's a great issue to discuss. Another reader submitted the following response:

Frustrated: is it possible that your standards are too high? Maybe this series of events has some other kind of takeaway. Three different editors have overridden your negative reports ... perhaps your standards are either too stringent or just out of keeping with what the editors are looking to publish. I've had an editor override a referee on one of my papers. The referee accused me of all kinds of weird things (like I'm *clearly* just a graduate student, and some senior scholar has made some edits to my paper) and seemed really irritated. The editor contacted me with this review and asked if I would make some changes. The paper was then sent out to a third referee, since the editor judged the second referee to have an overly harsh view of the paper. I was grateful to the angry referee despite all the mean-spirited commentary, since I was able to improve the paper. I am not an established scholar, nor is anyone running around wielding influence on my behalf. Did the editor act inappropriately? I don't think so, not at all. These are recommendations you are giving, not dictates.

This is more or less the reaction that I had to Frustrated's comment. If I am reading Frustrated's comment correctly, in just the past few months they have had three (different?) editors (at three different top-ranked journals?) ignore their negative referee reports in favor of more positive reports by other referees. What should Frustrated infer from this? While it's hard to know for sure without looking at the papers themselves, four things seem to me worth noting here:

1. Referees and editors at top-ranked journals tend to have very high standards (viz. high rejection rates).
2. In each of the cases the OP describes, at least one reviewer thought quite highly of the 'really really bad' paper in question, judging the paper to warrant only minor revisions.
3. The associate editors who handle a paper may be specialists in the area.
4. When they receive conflicting reports, these editors may read the paper themselves to decide which report(s) to trust.

Given these four things and given the fact that same thing has happened to Frustrated repeatedly, it seems to me that Frustrated should seriously consider that the possibility that (A) they are writing overly negative referee reports, and (B) that the papers they judge to be obviously really really bad aren't. After all, if the papers really were that bad, one would figure that someone else--either the other referees or the editors who read the reports and then the paper--would see this as well. But that's not what's happening here. In three separate cases, it seems, the editors have read the referee reports and the paper and decided for themselves that the other referees' positive reports are the ones to trust. 

Now, this read on things could be totally wrong. Maybe Frustrated is absolutely right about these papers, and in each case the referees and editors have gotten things wrong. Anything is possible! However, if I were in Frustrated's position, I guess I'd think the appropriate thing for me to do would be to adopt a stance of epistemic humility and infer something like the above explanation. Indeed, I think I'd be especially inclined to adopt this perspective given what is known about peer-review in general: namely, that referees and editors already appear to have a documented tendency to reject great papers, judging them not to be great when they are. After all, consider all of the Nobel Prize winning economics papers that were famously rejected from top journals in that field on grounds (such as referees claiming their results to be trivial) that, in retrospect, appear to be ridiculous. Or consider Jason Stanley's report from a few years back that many of his papers that were rejected from multiple top-ranked philosophy journals are now among the most cited papers in those very journals. Or consider Heesen and Bright's discussion of the empirical literature on peer review, which finds (among other things) that referees agree on the quality of papers barely better than pure chance. All of these things suggest to me that when we are serving as reviewers, we should adopt a standpoint of epistemic humility--and so if I found myself in Frustrated's position, I guess I'd think that the appropriate thing for me to do is to rethink what I'm doing as a reviewer. 

But these are just my thoughts. What are yours?

Comments

[aphilosopher]

I think we all can agree that plenty of crappy papers get into top journals, seemingly because they're fashionable or written by someone in the right circles. So, without seeing the papers or knowing any other details, there's some chance Frustrated isn't the problem.

But, I agree Frustrated should do some honest, humble reflection on their reports --- e.g., the bar for acceptance shouldn't be whether you're "convinced" or the argument succeeds, it should be whether this is an interesting paper that's worth discussing and makes a contribution to the literature.

[Michel]

I agree: given that the two referee reports diverged significantly, I would just assume the editor took a look and decided my verdict was too harsh (or generous, as the case may be).

[Tom2]

"Frustrated" confuses me. If he/she thought that the paper was "really really bad", why would he/she recommend revision? (Especially since we're talking about a "top journal" here?)

Personally, I don't care what editors decide to do in light of my referee reports. Their job is to select papers for their journal. A critical criterion (albeit not the only one) is paper quality. It the editors' job, and their responsibility, to judge paper quality.

My job as a referee is to provide advice on this (and other issues) and expert insight. In a way, I'm happy to see behavior like what "Frustrated" describes, because it means that journal editors are doing their jobs, and not pushing off their responsibilities on referees.

[Frustrated]

Thanks for all your comments!
Tom2: I recommended major revisions because the topics and theses were interesting. Given also this covid situation, I understand that many of us are struggling, so if the ideas are good I give another shot.
Anyway, all comments are extremely useful - I'll reconsider my strategies as a reviewer (there is indeed a pattern!). But I want to emphasize that I always try to give constructive comments when I'm asked to review a paper (and in general I received constructive comments on my papers). Thanks!

[Marcus Arvan]

Hi Frustrated: Thanks for chiming in! I'm glad that you always provide constructive comments and that you're willing to reconsider your refereeing strategies.

On that note, I guess I'll just point out that I find something about your overall description of the case puzzling. In the original comment, you say that the paper is 'really really bad'. But here you say that 'if the ideas are good I give another shot.' This suggests that the paper *isn't* really really bad after all (which is what the other referee and journal editor evidently thought). So it sounds like you may be holding papers to too high of a standard for publication.

I don't know if this describes your approach to reviewing, but some reviewers seem to think that to be publishable, a paper must be absolutely 'watertight' and respond to every possible objection a reader might have. If this is your approach as a referee, then I could see why you might think that a paper with 'good ideas' might not be worth publishing. But I don't think it's a good standard to use. The test for publication should be: does the paper make a valuable and interesting contribution to the literature? A paper can do this *without* being watertight or responding to every objection. Just ask Kant. The Groundwork was well worth publishing. It also happens to have holes all over the place that people have objected to and tried to fill for over 200 years.

If a paper has good ideas and makes a worthy contribution to the literature, it should be published. As reviewers, we shouldn't hold papers to (most likely idiosyncratic) standards of perfection!

[Frustrated]

Thanks Marcus. What you say is indeed an important lesson

[Tim]

I'm confused by the description of the case. Was Frustrated asked to review the same paper at three different journals? And then, at the third journal, Frustrated asked for revisions and it was ultimately accepted? If that's the case, then it seems like twice Frustrated rejected the paper and twice it was. Once, Frustrate recommended revisions, and that was the verdict. And only once did Frustrated recommend rejection and that was not the verdict. So 75% of the time editors agreed with Frustrated. That doesn't seem so bad. But maybe I didn't understand the case.

Regardless, I second Tom2's comment. This may be a moment to reflect on one's reviewing habits and exercise some epistemic humility. Alternatively, this may be a moment to reflect and become less emotionally invested in one's reviewing. Journals are a mess and its generally wise to not become emotionally invested in messes.

[Overseas Tenured]

This happened to me a few times, both as a reviewer and as an author. Other than the editor needing to decide which referee to trust, there's also the issue of inconsistent referee reports. So on one occasion, the editor of a top journal didn't merely give me an R&R in the face of a report that recommended major R&R and another that recommended rejection, but explicitly asked me to ignore the second report, probably because it was impossible to rewrite the paper in a way that takes both of them into consideration. (The second report actually wasn't bad at all; but it did demand revisions that would have required me to take the paper in a direction that I didn't want to.)

As a referee, I don't take it too much to heart when this happens. However, if a journal repeatedly and consistently does this, I'll probably decline reviewing for them in the long run because I'll assume that I'm just being used as a rubber stamp.