What Do You Aim for in Referee Reports? (guest post by Finnur Dellsén)

This is a guest post by Finnur Dellsén, Associate Professor at the University of Iceland

As many people have pointed out recently, the peer review system relies on us accepting several referee requests for each paper we publish. So refereeing is a substantial part of our job as researchers, even if we don't get paid for it specifically. But how do we make sure we write helpful, insightful, and constructive referee reports? How do we improve as referees? We get all sorts of feedback on our own work --- from colleagues, editors and referees --- which helps us improve as researchers. We get much less, if any, feedback on our referee reports.

So when I was named Referee of the Year at the British Journal for the Philosophy of Science for 2020, I thought it would be a good opportunity to get the conversation going on how to write a good referee report (see my Facebook post). Below is a list of things that I aim to do in my reports (but I encourage others to add to the list in comments):

1. I always summarize the paper first in my own words. This lets the authors and editors know I carefully read and understood the paper. It also forces me to focus on the main contribution of the paper as opposed to minor concerns (which is important because of point 3 below).

2. I try to find as many nice things to say about the paper as I possibly can, especially when I recommend rejection. I highlight ideas and arguments that are especially interesting or compelling. This helps the author know what not to change when revising their paper (e.g. for another journal).

3. I try to make the amount of text in the review that I spend on each idea in the paper reflect its importance for the main thread of the paper. Every paper has things in it that I don’t agree with or find slightly unconvincing, but it’s usually not worth getting bogged down with those issues.

4. When suggesting changes, I say explicitly which of those changes I consider to be most important, even dealbreakers, and which ones are mere recommendations for the author to consider. This makes it clear to the author which changes to focus on.

5. I try to indicate clearly, if not to the author then at least to the editors, how confident I am in my recommendations -- not just about whether to publish, but also about individual points or recommendations for changes. In many cases, I am not an expert on all of the things the paper discusses, and when I’m not I try to say so explicitly.

6. Finally, referees can gently point to relevant work by underrepresented groups and/or work that is underrecognized in the field (e.g. because of prestige bias). I admit I haven‘t done this as often as I would have liked in the past, but I think it can do a lot to increase diversity and make philosophy more inclusive.

Those are some points that come to mind. What am I forgetting? What do you aim for in your referee reports?

(Thanks to Helen De Cruz for giving me this opportunity to discuss refereeing practices, and to the BJPS for the award!)

Comments

[Tim]

I'm glad this topic is being brought up. I also do 1, 3, and 4. I do not "find as many nice things to say about the paper as I possibly can." However, I do normally list a sentence or two about what I think works well. 6 also seems like an ideal worth aspiring to.

However, given my verdict, I think of my audience differently. If I am recommending rejection, I think of my audience as the editors and my task is to justify my decision. I don't think of my task as helping to improve the paper. If I am recommending acceptance or 'revise and resubmit,' I think of my audience as also including the author and I do make recommendations (or requirements) for improvement. I wonder if this practice is common.

[Neil Levy]

This more or less reflects my practice with the exception of (3), and I would hate to see (3) established as a convention (such that if the reviewer devotes little attention to an idea, we are justified in inferring she thinks that idea plays a small role in the paper. That might be a reason to think she hasn't understood it and to downgrade her report).

My report will focus on the problems with the paper, and identify ways they might be fixed. That entails that huge tracts of the paper might not receive any attention. If I think they're fine as they stand, I won't mention them (except insofar as they might be mentioned in the first para, summarizing the overall argument). Time and space are limited, and I certainly don't want to waste my time and the author's time on repeating the good parts of the paper at sufficient length to ensure its place in my report matches its place in her paper.