How to write a referee report (John Greco)

This guest post is by John Greco, currently Leonard and Elizabeth Eslick Chair in Philosophy at Saint Louis University, and soon in McDevitt Chair in Philosophy at Georgetown University 

I am sure that there will be varying opinions about how to write a referee report. In keeping with the spirit of this series, I here offer my own opinion, based on my own experience as someone who has written and read quite a few such reports. I have written them as a referee, of course. I have read some as an author, but many more as the editor of a major philosophy journal. It is this last perspective, I believe, that is most useful for present purposes. For referee reports, remember, are written primarily for journal editors.[1] That is, the defining purpose of a referee report—its raison d'etre—is to help a journal editor to make a decision about a submission. And that is what gives us insight into the criteria for a good referee report.

Already I have said something (at least one thing) that is substantive and controversial. That is, I don’t think that everyone would agree that this is indeed the primary purpose of a referee report. Many writers of such reports seem to think (or at least this is what I gather from reading their reports) that referee reports are primarily directed at authors, and for the purpose of improving the paper that is being evaluated. In my opinion, this is a mistake, and it leads to many of the problems that are typical of referee reports. In any case, below I will describe what I think are the essential elements of a good referee report, guided by my understanding that referee reports are supposed to help editors make a decision on a submission. Following that, I will go over what I take to be some common mistakes and unhelpful practices.

The elements of a good report.

In my opinion, the best referee reports have a few essential elements.

First, a good report contains a very short précis of the paper being evaluated, including

a) the thesis of the paper

b) the author’s strategy for establishing the thesis, and

c) some commentary on why the paper is important, interesting, timely, etc.

This may or may not include d) a very broad outline of the paper.

Again, all this should be done in very short order—one or two sentences on each point. This is useful because it directs the editor to these essential elements and characteristics of the paper. But it also signals to the editor that the referee has understood and appreciated these essentials. It also demonstrates that the referee has the expertise needed to communicate them in clear and efficient fashion. In sum, a clear and efficient summary of the paper and its importance both informs the editor and builds confidence in the referee. Moreover, this kind of summary signals to the author that her paper has been understood and appreciated. And if there are some mistakes or misunderstandings in the report here, the author will now be in a position to refer back to them specifically in communications with the editor.

Second, a good referee report will give a clear summary of the reasons for the referee’s recommendation. That might seem too obvious to mention, but many reports fail to do this. For example, many reports will list a series of criticisms and/or virtues of the paper, but make no mention of the weight or importance of these various considerations. So, all the editor gets is a series of pros and cons, but no explanation regarding why these add up to “reject,” “accept,” “revise and resubmit,” etc. The better alternative would be to summarize what are the deciding factors among the various considerations being offered—what are the deal breakers (or deal makers) that are driving the referee’s recommendation. Put differently, a good referee report justifies the recommendation being offered, and does so in clear and efficient fashion.

Third, and in keeping with the previous theme, a good referee report will in some way flag more important vs. less important considerations as they occur. For example, a good report might be divided into the following sections: Summary; recommendation; major considerations; minor considerations; further comments for the author. This last “comments for the author” section can be used for comments that the referee thinks might be helpful to the author, but that should not factor into a recommendation for or against publication. This last point is related to my introductory remarks above, where I suggested that the primary purpose of a referee report is to help the editor make a decision. That is consistent with helpful comments directed at the author. But such comments are appropriately relegated to a section of the report, and clearly flagged as intended for such purpose.

Finally, the summary and recommendation parts of a report should be short and focused. That is, the editor should be able to understand the paper and the reasons for the recommendation fairly quickly. After that, a referee might go into detailed commentary on various virtues and vices of the paper, and might provide additional detailed comments aimed at helping the author. But all this is secondary to the guiding purpose at hand, which is to help the editor with his or her decision.

Some common mistakes and unhelpful practices.

Clearly enough, some referee reports are unhelpful because they fail to include one or another of the essential elements discussed above. I will leave it to the reader to consider the variations on how this might happen. In the remainder of the post, I will discuss two other common mistakes that referees make and that result in their reports being unhelpful. Both involve the recommendation of “revise and resubmit.”

First, referees often recommend “revise and resubmit” when they should recommend “reject.”

I won’t speculate why, but some people have a very hard time rejecting a paper outright. For example, as editor of APQ, it was not uncommon for me to receive reports that listed more than one reason sufficient for rejection, but then recommended revise and resubmit. Consider the costs of this. First, the author is now encouraged to revise a paper with the implication that the requested revisions will make the paper publishable. Second, that revised paper will now have to be refereed again, and by more people than just the referee recommending revisions. This makes more work for an already overloaded system. If there is no practical possibility that the resubmitted paper will now be publishable, the referee has done a disservice to everyone involved, author included.

Second, referees often recommend “revise and resubmit” when they should recommend “accept.”

Here I will speculate on the referee’s frame of mind. I believe that, at least in many cases, the referee has lost sight of his or her primary responsibility, which is to help the editor to decide whether the paper is worthy of publication, and not whether it could be improved in some way. What is more, in my experience as editor, recommended revisions would often have detracted from the quality of the paper, by insisting that some implausible objection be considered, for example, or by insisting that some further, tangential issue be explored. If you wonder why philosophy papers nowadays are often tedious, here is one major factor in my opinion.

Notice that in both cases (where the paper should have been rejected or should have been accepted) the referee could have included his or her comments in a “for the author” section. That is, even in a case of rejection, the referee could still suggest revisions for a possible descendant of the paper. And even in a case of acceptance, the referee could suggest revisions that she thinks would further improve an already publishable paper. But in either case the verdict of “revise and resubmit” is inappropriate, and oversteps the responsibilities of the referee.

Of course, it is ultimately the editor’s responsibility to decide whether a referee’s recommendation should be accepted, or whether a proposed revision is reasonable in that context. But the referees for a paper are usually more expert than the editor on any given paper topic, and so editors will rightfully defer to their recommendations in most cases. For that reason, referees should keep their primary responsibility clearly in mind, and write their reports accordingly. If you do, your reports will be appreciated by everyone involved.

 

[1] Actually, referee reports can be directed at other kinds of editor as well, such as a book series editor, or at some other person who has the role of accepting or rejecting a submission, such as members of a grant application committee. In this post, I will talk in terms of referee reports for journal submissions, but much of what I say would carry over to other kinds of referee report as well.

Comments

[Paul]

This is a very informative post, thanks!

[Anon]

Thank you! This is so helpful.

[Tom Cochrane]

Hi John
Have you ever considered giving credit for good referees, e.g. by awarding points that can be displayed on your website? This could be both very motivating, and helpful as feedback.

[John Greco]

Hi Tom. APQ does not do that, and I am not sure I like the idea of a point system. One of my colleagues recently said, in a different context, that "I don't work for points," and I understand the sentiment. But I am sure that, more generally, it is a good idea for journals and for the profession in general to figure out ways to recognize and reward good referee work. It is amazing how much time and effort goes into writing good referee reports, and how little recognition there is for it.

[John Greco]

. . . in fact, the only current reward for good referee work is more referee work!

[Chris]

BJPS has now started a "referee of the year".

See
http://www.thebsps.org/2019/03/referee-of-the-year-2018/

I don't know if it comes with any money, though.

[Craig]

This is great -- very helpful. But I'm curious about the skepticism about revise-and-resubmit, the most common verdict for me to return. I often find myself thinking both "I would not recommend publishing this paper" but that "I can imagine that the author could change my mind with significant revisions" and "I would definitely be willing to read this again." (In fact, I don't think I've ever had an initial reaction other than that combination.)

One possibility is that I'm just too stingy with "Accept." That's very believable. But often I think something like, "This paper does not sufficiently engage with the relevant literature" or "I would not say to colleagues working on this topic, 'Here's a paper that would be helpful to you'" or "It is hard to imagine further scholars taking this paper up for criticism or endorsement, beyond maybe showing up in a string citation of papers-on-a-topic." So I feel comfortable with my disinclination to accept.

But then should I 'reject'? Maybe. I will reject a paper if it does nothing other than make a point already extant in the literature, if it is particularly poorly argued, or some similarly dismal verdict. But maybe I'm too optimistic about the possibility of improvement?

And I wonder about the claim that r&r creates more work? Presumably the author will often just march off and submit somewhere else, so doesn't it just move the work? Maybe some authors will give up, so there's some effect.

Again, thanks for the helpful post, lots to think about!

[Casey]

Great post. Thanks for this!