Reviewing a bad paper for a journal?

In our newest "how can we help you?" thread, a reader writes:

For the first time is my (relatively short) career, I have been asked to review a paper that is very bad.

We all know the 'Reviewer 2' meme, and have probably heard of the research about the damaging effects overly that negative reviews can have on people - and I don't want to be that reviewer!

So, I'd love some advice focused specifically on how to write negative journal reviews without being too harsh in the tone. How does one best communicate a negative verdict without going over the line?

This is an excellent question, and I'd love to hear readers' answers. Although sometimes it can be hard to avoid being 'reviewer 2' (particularly if, as in the OP's case, the paper one is reviewing seems really bad), here are a few tips that I have found to be useful:

1. Use neutral, non-inflammatory language: As this comparative study found, something like 30% of reviews of philosophy papers contain inflammatory rhetoric. To avoid doing this as a reviewer, I've found that it can be useful to write one's review and then read through it multiple times to see whether there are any unnecessarily hostile words or phrases that I might replace with more neutral ones. 

2. Focus on detailed, dispassionate commentary rather than 'meta-commentary': Is the paper's argument confused? Instead of just saying it is confused (a meta-commentary), along with perhaps a short toss-off comment on why ("The paper commits all kinds of fallacies"), focus instead on showing the error(s) in question. Say, for example, "On p. 1, the paper claims X. On p. 2, it claims to derive Y from X. Yet Y does not follow. Here's why...". My experience as an author is that when one receives these kinds of explanations, a reviewer report can at least feel fair-minded--whereas if report merely says "the paper is confused", it can come across as unfair and needlessly hostile.

3. Set the paper aside for a week or so: Although as a reviewer one should presumably avoid unnecessary delays in completing one's review, it's amazing in my experience what a bit of 'distance' from a review can do. When you read your own review 'afresh', as it were, things can just pop out at you (like, "Wow...yeah, I should really tone that down").

4. Try to say a few laudatory things: Although it may seem hard to do in cases of especially bad papers, my experience is that papers that pass the desk rejection stage usually have at least some virtues. So, if there are any, mention them, particularly toward the beginning of the review.

5. Remind yourself of reviewer fallibility: I assume you have received referee reports that you thought were unfair, right? Well, those reviewers were presumably someone like you: someone who thought they were reading a 'really bad' paper. For this reason, I think it's helpful to consciously remind oneself of just how fallible we can be as referees. Try reminding yourself of this study, for example, which found a high number of classic (and 15 Nobel prize-winning!) economics papers to be rejected from journals, often for the most spurious of reasons (such as being 'trivial'). Or remind yourself of this story by Jason Stanley. Personally, I've found that reminding myself of how fallible we can be can really help one tone down a review. 

6. Read this discussion of bad reviewers by Michael Huemer: Don't reject a paper simply because you disagree with it, or even ultimately find the argument unconvincing. Your task as a reviewer should not be to expect perfection--especially given that, as we all know from graduate school, even the 'most important' papers in the field are usually flawed (or at least dubious) in all kinds of ways. Ask yourself instead, 'Is the paper interesting enough and good enough to publish, or at least give an R&R to?'

7. Imagine receiving the review yourself: I've heard people say they dread hearing from journals, and I empathize. Like most people I've talked to, I've received my share of harsh, meanspirited reviews. So, before submitting your review, ask yourself what it would be like to receive your review as an author. If your review feels needlessly 'hard to read', try to adjust its tone.

Anyway, these are just a few thoughts. Sometimes they can be admittedly hard to follow through on, particularly when one receives a paper to review that just seems incompetent in some way or other. Chances are, many of us will be 'reviewer 2' at some point in our lives. Nobody's perfect. But it is worth trying to do one's best to avoid it. Anyway, what do you all think? Do you have any good tips for avoiding being reviewer 2? If you do, I'm really curious to hear them!

Comments

[PT]

I'm going to have to respectfully disagree with Marcus on this one. There's nothing wrong with saying that a paper is quite bad, and in fact I think you're obligated to do it.

First off, referees don't owe any obligation to the person whose paper they are reporting on. Instead, the referees' obligations are towards the journal and towards the profession at large. (If you disagree with the last disjunct, consider that "X does not make a novel contribution to the literature" is a good reason to reject paper X.)

I think this means that if a paper genuinely is quite bad, it is the referees' obligation to communicate this to the editors (and, if the editors deem it necessary, they'll pass it on to the author). I can't imagine thinking you have ANY obligation to say anything nice or "laudatory" about the paper. Moreover, if we think that referees have obligations to the profession at large, then if the paper really is very bad, it should be one's obligation as a referee to communicate this to the author, too, since it'll bring it to their attention that their work needs to be improved.

Also, re: Marcus's "avoiding being Referee #2" comment: I agree that we should avoid being Referee #2, but the "Referee #2" stereotype is that of an irresponsible referee who does not actually read the paper and leaves unhelpful comments. It is perfectly possible to be a responsible referee, and to leave helpful comments, if the paper is genuinely quite bad and the only helpful comment is: "This paper is bad and it should not be published." (Think of it this way: if a student's paper warranted a grade of a D or an F, you wouldn't think that it would be *irresponsible* to give it a bad grade, no?)

[Marcus Arvan]

Hey PT: To be clear, I am not suggesting that referees should avoid an honest evaluation of the paper. I often write critical reviews (and always honest ones). The real point, or so I think, is how one does it: snidely or respectfully.

Also, there have indeed been times where I haven't been able to find anything good to say about a particular paper. But, I think, this is pretty rare. For the most part, we are evaluating papers by professionals--people with many years (if not decades) of high-level training. If one regularly finds oneself having nothing good to say about papers as a reviewer, then I think that probably says more about the reviewer than the papers they are reviewing.

That being said, we evidently do disagree about something fairly fundamental here. You write, "referees don't owe any obligation to the person whose paper they are reporting on." I respectfully disagree. We owe everyone with whom we interact with decency and respect (the sole exception perhaps being when people behave so odiously that their behavior warrants the contrary). Not that we (or I) always live up to this standard--but, for all that, I think it is one worth striving to meet in all areas of human life.

And indeed, to address your student paper case, sometimes I do give D's or F's. That being said, as a rule, I have learned it to be better to give students with constructive rather than meanspirited or snide feedback. The D or F (and dispassionate justifications given for it) should suffice. The same, I think, should go for reviewing.

[anon]

"There's nothing wrong with saying that a paper is quite bad, and in fact I think you're obligated to do it."

Maybe I'm taking you more literally than you want to be taken, but "This paper is quite bad because [...]." is something I'd never write in a referee report. I would say instead, "My recommendation is 'reject' because of these three problems with the paper: [...]."

[Chivers Butler]

It might be helpful to distinguish two questions:

1) What may/should be said in an author-facing report?

2) What may/should be said in a report for the editor only?

My sense is that one can and should be quite blunt in the second case.

[question for op]

Shouldn't we be asking the OP about the respects in which the paper is bad? Is it poorly written? Rushed? Leaves very obvious counter-examples untouched? Doesn't cite those who they are academically and intellectually obligated to cite?

Or, do they mean by "bad": "I don't agree." The whole problem with bad reviews is that sometimes, that is what is going on. So what makes a paper this bad?

[PT]

First commenter here, replying to Marcus and others.

@Marcus: Obviously I agree that we owe basic norms of courtesy and respect. Likewise, I never said that one should give their students meanspirited or snide feedback. I don't think what I said could be reasonably construed as saying that referees or teachers can just be jerks simpliciter. I was just using a quantifier restriction.

All I meant is that *in their capacity as a referee*, a referee is under no obligation to make the author of a paper feel good about their paper, or to write a report that avoids being direct and honest about the (perceived) merits or faults of the paper. For instance, I was disagreeing with the suggestion that referees should "try to say a few laudatory things" about papers that are very bad.

Additionally, I wasn't suggesting that the OP (or myself, or anyone else) rarely finds themselves having nice things to say about a paper. But the OP specifically asked about a paper that is "very bad". I take this to mean something like the paper itself is just very unclearly written, or makes egregious mistakes concerning the established literature, or makes flat out factually incorrect claims. I think when these are the circumstances, it's better to be honest, and to say, e.g., "Look, this paper is just very unclearly written and the quality of writing in it just does not meet professional standards." I don't think that's meanspirited or snide, and in fact it seems to me to be well within what is expected of a responsible referee. On the rare occasion when I am asked to referee a paper that turns out to be very unclearly written, or which contains factual errors, or egregious misunderstandings of the literature, that is in fact what I write in my reports. I think that's warranted.

@Anon: Yes, that's exactly right. On the occasion when one has to referee a paper that is just really, painfully unclearly written, or which contains really bad misunderstandings of what's going on in some literature, I think it's quite appropriate to basically say: This paper should be rejected bc its writing is not up to professional standards / ...it badly misunderstands what is going on in the literature.

[Marcus Arvan]

Hey PT: I wasn't suggesting in the OP that we try to make authors feel good. The tips that I was offering are intended merely as rules of thumb to combat the (very real and pernicious) tendency of (some) reviewers to go overboard to (unnecessarily) make authors feel bad. I still think they are good rules of thumb, including the thing about trying to say something positive.

Why? Well, see the study that I link to in the OP on how common inflammatory comments are in philosophy reviewer reports. Or reflect on some of the reviewer reports you've received (since virtually everyone I know mentions receiving reviewer reports that seem unfair and needlessly inflammatory to them!). Finally, if you haven't read it, I really recommend reading the piece that I linked to in the OP on rejected Nobel-prize winning economics papers. It's a proverbial eye opener.

When academic journal reviewers have a documented tendency to reject future *Nobel-prize* winning papers as "trivial" or (I kid you not, this is a real one) "This amounts to nothing and should be refused", there's a problem, right? The problem isn't that referees are too honest. It's that sometimes they needlessly (and erroneously) go out of their way to make authors feel terrible about themselves. So, or so I'm inclined to think, we should adopt strategies to combat these tendencies.

Sometimes, there really is nothing positive to say about a paper. But, the vast majority of the time, my sense is that there is *something* good about a paper if it pasts desk-review--so the tip there (to try to say something laudatory) is, I think, a good rule of thumb to try to bear in mind.

Anyway, that's what I took the OP to be asking for tips about when they mention "the research about the damaging effects overly that negative reviews can have on people - and I don't want to be that reviewer!" They were merely asking how to not be a jerk ("How does one best communicate a negative verdict without going over the line?"). So, if you agree that we shouldn't be jerks, we're on the same page--though here again, I'm with anon: it's all about how you say what you say.

[Anon UK Grad]

Op here, answering 'Questions for OP'.

I don't just mean that I disagree with the paper (though that also happens to be true) - and, for what it is worth, I have happily recommended acceptance for papers that I disagree with in the past. Refereeing, in my view, is not about enforcing my own views, but rather about ensuring suitable quality.

But, in any case, I don't want to say much about the specifics of the paper - I am hoping the discussion can be general enough to be helpful to others who find themselves in this situation as well.

[Tim]

OP wants advice regarding "how to write negative journal reviews without being too harsh in the tone." From one perspective, this is impossible. A report where you reject a paper is *always* harsh in tone--its a rejection! Beyond that, I think the recommendation of neutral/non-inflammatory language is best. Its the most effective way of minimizing rudeness.

I'd also second the recommendation to divide out comments for author and comments for editor. I once was asked to reviewer a really bad paper. In my report, I wrote about 1000 words on the first three pages, detailing various problems in scholarship. But to the editor I said that I only stopped after three pages because I reached a 1000 words. I could have kept going for much longer on the rest of the paper. The author didn't need to hear that; it may have just upset them. But I thought it was important to communicate to the editor that my low opinion of the paper outstripped just what I had written. (I have even heard of some cases where Reviewers simply provide no comments for the author and tell the editor they don't even think the paper should have been circulated!)

[zebra]

I think the practice of writing separate comments to the editor is what leads to both abuse, and those getting their papers rejected thinking the system is not opaque. If you cannot say it in a way that the author of the (bad) paper should not see it, then do not say it. But really, do let the author know their paper does not meet professional standards.

[Prof L]

I would add: don't hypothesize about the author's career stage, gender, who has helped them with the paper or not, their nationality, their dissertation advisor, their level of education, their first or second language, their preparation or background, what they do not know, and so on. You're probably wrong, it's totally irrelevant, and can also get very nasty and personal.

I would say it goes without saying, but, well ...

Relatedly, "The author clearly does not understand X" or "The author knows nothing about Y" are speculative. Such things can be rephrased to be about what the paper demonstrates, or into a statement about why something that was not included should be included.

[Humility]

In dialing back earlier comments that appeared to defend excessively harsh criticism as sometimes deserved, PT says:

"I think it's quite appropriate to basically say: This paper should be rejected bc its writing is not up to professional standards / ...it badly misunderstands what is going on in the literature."

These kinds of comments are more boarderline, but for what its worth, I don't think you should be writing such comments either in a referee report.

Saying to a fellow professional "your writing isn't up to professional standards" is unnecessarily harsh. Better to say "the writing in this paper is unclear in these specific ways ... these shortcomings are significant enough for me to find this paper unsuitable for publication. [Aside: I've reviewed 30+ papers and although several have been poorly written, I've never had one written so badly that the writing alone was a reason for rejecting (as opposed to giving them a chance to correct it in an R&R). If you find yourself regularly rejecting papers because they are "poorly written" then the problem may be with your standards and mindset rather than with the papers themselves.]

Saying that someone "badly" misunderstands the relevant literature is also unnecessarily harsh. You should instead say "The paper misunderstands the relevant literature in these specific ways ... These misunderstandings are significant ones that undermine the papers relevance in this way ..."

Two general points to draw from this:

(1) Starting with broad sweeping statements about why the paper is bad or poor in certain basic ways often gives an overly negative tone. Better to start with milder statements "there is a problem with x" then give specifics about exactly what the problem is, and, only after these specifics have been given, explain why they are major flaws that mean the paper cannot be accepted.

(2) Understand that tone and color matter just as much as literal meaning. The sentence "this paper badly misunderstands x" may have roughly the same meaning as "There is a significant misunderstanding in this papers account of x" and "this paper misunderstands x in fundamental ways" but its tone is much harsher. On some psychological level that goes beyond semantics and pragmatics it says to the author "you have done a bad job, you have done badly here".

[Jonathan Ichikawa]

I just want to reiterate the key thing that Chivers Butler said, which is pretty central: you can and should use a very different tone to the editor than you do to the author.

You can say "here is why the author's project is interesting, and here are my reservations about accepting it for publication" to the author, and say "to be frank, this paper is just quite bad, and I don't see any reasonable prospect for improving it to publishable standards" to the editor.

[Michel]

The first paper I was ever asked to review was very bad. I wasn't entirely certain it wasn't a hoax paper. I said as much to the editor, but otherwise tried to engage with it genuinely, and left it serious, neutrally-expressed comments.

The only other time it's happened for me was refereeing a book. It was an absolutely dreadful piece of scholarship (to the extent it even was 'scholarship'). I did the same then as I did that first time.

That's twice in about ~30 sets of referee reports, so it's been quite rare (and I've never refereed a conference paper as bad as those, and I've refereed a lot more conference papers than articles and books!).