How to know if you're a good referee?

In our most recent "how can we help you?" thread, a reader writes:

I referee more and more for journals, and there is this question at the back of my head everytime I send a report: How do I know if I do a good job as a referee? Or: What are good signs that you do a good/bad job as a referee?

A bit of context: When I send my reviews to editors, I usually also send them a message saying "if you need more information, or if my report is unclear, please let me know". I never hear back from the editors. When authors get an R&R, they are not dismissive of my comments, but this is hardly good evidence (as an author, being nice is part of the game!). We've all had experiences with bad referees. How am I supposed to know I'm not one of them? Maybe getting new invitations to referee from the same editors is a good sign. But perhaps they don't even keep track of this.

Good questions! In my experience, one of the best indications of one's performance as referee are how one's reports compare to other referees' reports. First, some journals have the (very good) practice of sharing all referee reports when an editorial decision is rendered not only with the author, but with other reviewers. As a reviewer, I've often found this to be very eye opening. On some occasions, I've seen that my worries were shared by other referees--which gave me some independent evidence that I had raised the right concerns about the paper. On other occasions, my reports seemed better than the other reports, giving more detailed feedback. However, on still other (thankfully rare) occasions, other reports led me to wonder whether I did a good job with my report (as they either understood something in the paper better than I did, etc.). For these reasons, I think the practice of journals sharing referee reports with all of a paper's referees is a great practice, and I only wish more journals did it (as in my experience, very few do). Finally, of course, sometimes journals share other referees' comments (or the author's reply to them) when authors submit revised papers for revise-and-resubmit verdicts. I find this to be really helpful for the same reasons. Other than this, though, my sense is that all one really has to go by is one's own internal sense of whether one is doing a good job.

What do you all think? What indications do you think a referee can have that they are doing a good job? And, are there things that journals should do in order to give one a clearer picture of how good of a job one is doing?

Comments

[Douglas W. Portmore]

As an associate editor, the things that I most care about in order of their importance are: (1) you respond to requests to referee ASAP (and certainly within a day or two), (2) you get your report in on time, (3) you respond to emails from the managing editor immediately when you get emails telling you that your report is overdue and that we request an new estimate as to when we can expect your report, (4) your report makes a clear judgment as to whether the paper should be accepted as Is, conditionally accepted, given a revise-and-resubmit verdict, or rejected, (5) your report makes clear what the reasons are for that judgment, and (6) you provide some useful feedback for the author. I'd say if you do all these conscientiously, then you're an excellent referee. Unfortunately, far too many fail simply with regard to 1, 2, or 3. And some fail with respect to 4, 5, or 6. As far as knowing whether your reports are "good" and not just conscientious, keep in mind that if you were asked to referee, then your opinion is valued. So, if you are being asked to referee and you get your informed opinion along with your reasons for that opinion to me in a timely fashion, then you're an excellent referee. I rarely have a problem with the content of a report. Of course, that's not to say that I always agree with the referee's verdicts and the reasons for their verdict, but this doesn't mean that I think that the referee is bad. I asked them for their honest, informed, and reasoned opinion and if that what they gave me, then I'm very grateful. So, I think that it's wrong to worry whether your report is good in the sense of being the same thing as what others would say. If you're being asked to review, then your opinion is valued. All you need to worry about is being conscientious and providing your informed and reasoned opinion in a timely fashion while addressing any correspondence from the journal ASAP.

[Bob Hartman]

I like John Greco's advice to referees that was published as a blog post here and at the Daily Nous. I would recommend reading it!

In my view, a good referee report is a good argument. The conclusion is your recommendation to the editor. The reasons for your conclusion should be stated plainly and with clear support relations. And then you can stand back and ask the same questions you ask about your own papers: are my reasons lucid? Do they support the recommendation? How strong are those reasons, and do they adequately support the recommendation? Have I written these reasons charitably? (We are writing to other human persons to which we owe certain things.)

Whatever reasons you offer to help the editor make the decision will also be helpful to the author as they re-think their paper.

Good referee reports, then, are those that the referee can provide good answers to the above questions.

[ref]

Hi Marcus
I do not like the idea of journals sharing referee reports. These are supposed to be independent judgments of the paper. And seeing what another referee wrote undermines the independence, or at least threatens to.

[Chris]

Like Marcus, I like it when journals share referee reports. Since you only see the other referee's report after you've written yours, how does that undermine (or threaten to undermine) the independence of the reports?

[anon]

"When authors get an R&R, they are not dismissive of my comments, but this is hardly good evidence (as an author, being nice is part of the game!)".

Well, I think there's some evidence to be found for the quality of the OP's reports here. Many of my (successful) responses to R&R's involve arguments explaining why I think the referee made bad recommendations and why I won't follow them.

[Douglas W. Portmore]

Ref and Chris: Those journals that share reports (at least, all those that I know of) only share the reports after a final decision (acceptance or reject) on the paper has been reached. It's not just after the report has been turned it, as that could comprise independence were the manuscript revised and re-submitted. .

[in a (temporary?) non-academic position]

One thing I think to be important is to review the revisions you've recommended revision. And if it is already a revision, try not to reject a paper because it did what other reviewers requested.

Another thing is that, as have been stated in different places from my recollection, referee reports shouldn't be response pieces. I have recommended accepting papers that I deeply disagree with. The recommendation was due to my belief that the paper advances the discussion. I personally think having a different philosophical position by itself is an insufficient reason to reject a paper, but unfortunately, this sometimes happens.

[Evan]

(Wasn’t sure if this comment went through. But if it did, please disregard it)

How do you know you’re being a good referee?

A comparative approach is a fruitful epistemic means to arrive at such an answer as Marcus said.

However, a comparative approach can still be highly subjective since not every referee agrees what constitutes “good refereeing” in the first place. Whether or not you agree with Marcus really depends on what you consider to be the appropriate and inappropriate; necessary or unnecessary functions of refereeing.

There was once a post a while back outsourcing what different referees like or dislike in a paper and how they go about refereeing. The lesson to learn is that different refereeing methods will have different values depending on the person.

If you want an objective account of what is good refereeing vs. bad refereeing, then you need to figure out what the aim(s) or telos of refereeing or peer-reviewing is/are. Then the second step is figuring out what are the instrumental and appropriate means to achieving such aim(s) or telos.

We can even ask further: What do we mean by “good” here? Do we mean respectful, detailed-oriented, wise, effective, competent, generous, knowledgeable, fruitful, etc? What should referees be and do? And why? What shouldn’t they be and do and why?

I don’t have much of substantial answer because I’m not a referee. But I do hope to stimulate further reflection so that people can arrive at non-relativistic answers to these sorts of questions. A teleological approach may be more apt than a comparative approach in this case.