In our newest "how can we help you?" thread, a reader writes:
For the first time is my (relatively short) career, I have been asked to review a paper that is very bad.
We all know the 'Reviewer 2' meme, and have probably heard of the research about the damaging effects overly that negative reviews can have on people - and I don't want to be that reviewer!
So, I'd love some advice focused specifically on how to write negative journal reviews without being too harsh in the tone. How does one best communicate a negative verdict without going over the line?
This is an excellent question, and I'd love to hear readers' answers. Although sometimes it can be hard to avoid being 'reviewer 2' (particularly if, as in the OP's case, the paper one is reviewing seems really bad), here are a few tips that I have found to be useful:
1. Use neutral, non-inflammatory language: As this comparative study found, something like 30% of reviews of philosophy papers contain inflammatory rhetoric. To avoid doing this as a reviewer, I've found that it can be useful to write one's review and then read through it multiple times to see whether there are any unnecessarily hostile words or phrases that I might replace with more neutral ones.
2. Focus on detailed, dispassionate commentary rather than 'meta-commentary': Is the paper's argument confused? Instead of just saying it is confused (a meta-commentary), along with perhaps a short toss-off comment on why ("The paper commits all kinds of fallacies"), focus instead on showing the error(s) in question. Say, for example, "On p. 1, the paper claims X. On p. 2, it claims to derive Y from X. Yet Y does not follow. Here's why...". My experience as an author is that when one receives these kinds of explanations, a reviewer report can at least feel fair-minded--whereas if report merely says "the paper is confused", it can come across as unfair and needlessly hostile.
3. Set the paper aside for a week or so: Although as a reviewer one should presumably avoid unnecessary delays in completing one's review, it's amazing in my experience what a bit of 'distance' from a review can do. When you read your own review 'afresh', as it were, things can just pop out at you (like, "Wow...yeah, I should really tone that down").
4. Try to say a few laudatory things: Although it may seem hard to do in cases of especially bad papers, my experience is that papers that pass the desk rejection stage usually have at least some virtues. So, if there are any, mention them, particularly toward the beginning of the review.
5. Remind yourself of reviewer fallibility: I assume you have received referee reports that you thought were unfair, right? Well, those reviewers were presumably someone like you: someone who thought they were reading a 'really bad' paper. For this reason, I think it's helpful to consciously remind oneself of just how fallible we can be as referees. Try reminding yourself of this study, for example, which found a high number of classic (and 15 Nobel prize-winning!) economics papers to be rejected from journals, often for the most spurious of reasons (such as being 'trivial'). Or remind yourself of this story by Jason Stanley. Personally, I've found that reminding myself of how fallible we can be can really help one tone down a review.
6. Read this discussion of bad reviewers by Michael Huemer: Don't reject a paper simply because you disagree with it, or even ultimately find the argument unconvincing. Your task as a reviewer should not be to expect perfection--especially given that, as we all know from graduate school, even the 'most important' papers in the field are usually flawed (or at least dubious) in all kinds of ways. Ask yourself instead, 'Is the paper interesting enough and good enough to publish, or at least give an R&R to?'
7. Imagine receiving the review yourself: I've heard people say they dread hearing from journals, and I empathize. Like most people I've talked to, I've received my share of harsh, meanspirited reviews. So, before submitting your review, ask yourself what it would be like to receive your review as an author. If your review feels needlessly 'hard to read', try to adjust its tone.
Anyway, these are just a few thoughts. Sometimes they can be admittedly hard to follow through on, particularly when one receives a paper to review that just seems incompetent in some way or other. Chances are, many of us will be 'reviewer 2' at some point in our lives. Nobody's perfect. But it is worth trying to do one's best to avoid it. Anyway, what do you all think? Do you have any good tips for avoiding being reviewer 2? If you do, I'm really curious to hear them!
A question about journal etiquette. I have a paper that has been under review at a journal for nine months. For the past two months, I have not been able to get any response from the managing editor about the status of my paper. My question is: is it ok to write to one of the chief editors of the journal about this? Also, the journal is triple-blind, but as far as I understand that applies to the area editor who is handling the submission.
Good question! Another reader submitted the following reply:
I got a paper off the backlog at [redacted], where it had been stuck for nine months with no response from the ME, by emailing someone from the editorial board. I tried the chief editor first, but they had not responsive either. But the editorial board member was very apologetic and got me a decision in a week. Extreme times call for extreme measures, I think. My only caveat: be extra polite if the CE works in your area or it's a journal you need to publish in at some point (I purposely chose a board member as far removed from my area as possible).
I'm curious what other readers think. Do any of you have any helpful tips or experiences to share?