Writing good referee reports

A reader writes in:

Would it be possible to open a topic on your blog about advices for those of us who start to do reviews for the first time? I am in general on the other side of the publishing process: I submit articles that are often rejected (and quite often with some nasty comments). Now that I have to do reviews, I do not know how to be critical without being offensive.

Great query! I'd love to hear from readers, but here are a few tips that I've found helpful...

I always begin my referee reports with a brief introductory paragraph containing a brief overview of the author's thesis and argument (so the editor reading my report has some idea of what the paper is about), along with a clear editorial recommendation stated in language as neutral as possible. For example, if I think a paper should be rejected, I will write something to the effect of, "Unfortunately, I believe this paper should be rejected for reasons X and Y.' If I think a paper receive a revise-and-resubmit, I will write something like, "Although this paper has an interesting argument potentially publishable in Journal's pages, in my judgment it would require the major revisions described below." And so on.

Importantly, I try to keep the language here as neutral as possible, simply describing (in a couple of sentences) the substantive grounds for my recommendation (viz. "As I detail below, the author's argument is logically invalid"). I think this is important, because all too often I've received (and heard others receive) reports with needlessly inflammatory language, such as "This paper should have never been submitted to this journal" or "This paper looks like it was written by an undergraduate" (side-note: I know one influential senior academic who received a report stating the latter for a paper that went on to be widely recognized in their area!). Look, if a paper is really bad--and you think it was a waste of your time as a referee--I understand the temptation to let the author know in explicit language that theirs was The Worst Paper in the World. But I still think you shouldn't say things like this. If a paper is bad, a referee report written in neutral language (i.e. describing in philosophical terms where the paper errs) is sufficient to demonstrate that. Anything beyond that is simply hurtful--and, as I explain below, I find it helpful to remind myself of how it feels to be on the receiving end of this kind of stuff as an author. 

After my report's introductory paragraph, I always try to begin my report with something complementary. Sometimes, of course, it is hard to find something to complement the author for--and if the paper is bad from beginning to end, one may find it difficult (or even impossible) to begin a report this way. Indeed, I think I may have refereed a paper or two where there honestly wasn't anything positive I could say. That being said, I'm inclined to think it is good practice to try one's best to begin a report this way--not simply to go out of one's way to be "nice" to a paper's author, but because (in all honestly) most papers do (at least in my experience) have at least some redeeming features, at least if one looks hard enough. Since there are usually at least some good parts of a paper, I think it makes sense to recognize them at the outset of one's review. First, it seems to me intellectually honest, as in my experience academics can have a tendency to focus overwhelmingly on what's wrong with a paper instead of recognizing a paper's meritorious features. Reminding oneself to recognize merits first is, I think, a good way of counteracting the above tendency toward negativity. Second, my sense is that beginning one's review this way can be kind, as it can function (at least somewhat) to soften the blow of a report that is otherwise very critical. Although some academics may think it is "not our job" to soften the blow of our critical comments, I again find it helpful to remind myself of what it feels like to get comments that are entirely negative. I've heard more than a few people say that it makes them feel worthless, like a bad philosophers, and so on. Even if a paper is bad, my own conviction is that it is simply unkind to make other people feel that way (especially if, as is often the case, they have invested a lot of time and effort into the paper one considers bad).

Next, after saying something complementary, I provide the critical heart of my report: the part that functions to justify whichever editorial recommendation I made in the opening paragraph. In the case where I recommend acceptance, this is easy: I briefly explain why the paper is great. In cases where I recommend revise and resubmit, I begin (if I did not explain it in the former paragraph) explain why the paper is worth reconsidering, but why it needs revision. And, of course, in cases where I recommend rejection, I lay out in detail what I think warrants that recommendation. Importantly, when I recommend an R&R or rejection, I again try to write my critical comments in neutral language, simply explaining (descriptively) which parts of the paper appear to me to have serious problems (viz. "The author's first premise appears false. Allow me to explain."). In the process, I try to avoid needlessly colorful language (viz. "The author's first premise is absolutely absurd") and intensifiers (viz. "The author's writing is incredibly unclear"). One helpful tip I'd suggest here is composing one's review and then putting it away for a few days or a week before returning to it. My experience is that this can give one an important kind of "distance" from the review, such that when one reads it again one will experience it more like it was written by a stranger (and hence, experience it more like the author would). Oftentimes when I do this, some of the things in my report will strike me negatively--as too blunt, or needlessly aggressive, and so on. And so I will then go back in and try to temper the language, making it more neutral.

Finally, at the end of the day, before submitting my report, I try to ask myself, "How would I feel about receiving this report if I were the author?" As an author, I've always been happy to receive critical reports, provided they don't seem needlessly hurtful or 'personal.' Indeed, I've been happy to learn that some of my arguments don't work, and have given up on papers on occasion due to helpful critiques. There's nothing wrong with receiving a critical report, in my experience. It all depends on how the critical review is presented. And while I have not always been successful in this regard in my role as a referee (I would be lying if I said that I always lived up to the tips I've outlined above), my experience is that it is important to try one's best and that it can be rewarding to feel like one has been a "good referee" for the author and journal.

But these are just some of my thoughts. What are yours? Do you have any helpful tips for writing good referee reports?

Comments

[Ed]

My referee reports look a lot like Marcus's. But I'm wondering: Why do we bother with the introductory summary, which in some cases can be quite lengthy?

Do we include it to prove to the editor that we read the paper closely? No, editors don't check. To communicate the gist of the paper to the editor? No, editors picked up on the gist when they decided to send the paper our way initially. To communicate to the author that we read the paper closely? Perhaps, but then again, our subsequent feedback will communicate the same thing if we've really read the paper closely. To help ourselves in some way, e.g. remember the major moves in the paper? Maybe, though it's often not necessary if so, since we can usually consult our notes.

Any other ideas? Should we cut the summaries?

[another summarizer]

I include a summary so the author can see what some other person reading their paper thinks they are writing about. If the author does not recognize the summary as their key point, then they have a problem. They are not reaching their audience on the most basic level. I also, sometimes, recommend that the author only address part of what they currently address. That is, if I am recommending revisions I might refer back to the summary and say the author should concentrate on (A) and drop (B), given that their arguments for (B) are weak, or the really original contribution is (A).

[Noah]

There's also this from the (now abandoned?) Notes from the Editors blog:

https://fromtheeditors.wordpress.com/2013/12/28/what-makes-a-good-referee-report/

[Michel]

I agree with everything Marcus says.

On summaries: I actually find that writing the summary helps me take a few steps back, and be much more cordial in my tone. In particular, writing the summary helps me to get a better sense of just how clear I think the author's presentation was. And it helps me keep (what I take to be) the author's goals in mind as I'm writing my report. The crucial question, I think, should always be whether they've met those goals.

One thing I'd add is that I've found, as an author, that the best comments are those which suggest potential avenues of response to the referee's concern/objection, especially if it's what sounds like a fairly devastating objection, or if it touches on material that I don't know very well (my best experience concerned some finer points of logic, actually). It gives me a clear place to start as I'm reworking the paper, and it helps me orient myself vis à vis the referee's line of thinking. So: when I generate reports, I try to return the favour by suggesting what I think might be fruitful avenues of response or exploration in relation to what I take to be my most important objections.

[anon]

Following up on Michel's point: even if you don't want to go as far as suggesting how to respond to the objection, please explain clearly what your objection amounts to. I've had several referee reports for R&Rs that were mostly a list of angry questions, that implied objections without stating them explicitly. This made the it hard to figure out what the objection really was, or what a satisfying revision would really look like.

[Steven French]

Might I also suggest this, from Beth Hannon the inestimable Assistant Editor at the BJPS:

http://www.thebsps.org/2015/03/howtoreferee3/