Deciding whether to review a paper?

In our newest "how can we help you?" thread, an early grad student writes:

I'm wondering how to decide whether to referee a paper. I just got my first request, and from what I can gather from the abstract, this doesn't seem like an editorial call I'm particularly qualified to make. Would appreciate any thoughts.

This is a great query. Another reader (an editor) submitted the following reply:

Do not agree to referee a paper you believe you are not qualified to referee. Referees are to give their expert opinion - if you are not an expert, then your opinion is irrelevant.

I think this seems right as far as it goes (if a bit blunt!), but I'm curious whether we might be able to give the OP more helpful positive advice. How do you decide whether to review a paper? 

Comments

[EFB]

I take a few things into consideration.

1) Do I know enough about this area to make a qualified judgemetn (obvious).

2) Do I have time right now to do it within a few weeks?

3) Have I already 'done my bit' (I usually try to review at least as many times as I've submitted)?

4) Even if 3 is 'yes', if I am particularly interested in the paper, I'll usually review it anyway.

FWIW I say yes to most requests. I am early career so only get asked about 5-6 times a year. The ones I've turned down have either been too far out of my area of expertise or because I know the identity of the author.

[jakub]

I sometimes write back to the editor and ask why they came across me. There may be a reason for asking me for a review that cannot be disclosed in advance. This is usually the case when the title and abstract are too general.

[Assistant Professor]

EFB's points all seem spot on. We often hear to "review as many papers as you submit" but as early career you usually don't get many reviewing requests so it can be hard to get started reviewing. But my experience is once you publish on a topic then journal editors will find you and ask you to review papers they see as related to your published work (especially if the paper under review cites you).

But are we "experts" on every topic we publish on? Not necessarily. But you know enough about the topic to have your own paper successfully pass peer review on the matter. One thing I think reviewing other people's papers helps me keep in mind is that peer review is just that: done by *peers*. So while I agree that reviewers should have relevant knowledge to review a paper (including knowledge about what does and does not constitute a constructive review!), I think whether we construe this knowledge as "expertise" is complicated. Not every paper on a topic is reviewed by a top expert on that topic - there is just not a sufficient volume of experts/reviewers for this to happen. I think recognizing this should add some necessary perspective and humility to the peer review process for both authors and reviewers.

[non-TT]

I agree with the points above, and particularly those by Assistant Professor.

I would also add that there is always a reason why editors ask the reviewers that they do. When I get a request to review a manuscript on a subject outside of my own research wheelhouse (e.g., in the historical era in which I work but on a subject the literature of which I do not know), I assume it is because the journal has exhausted their options for experts on that particular subject. I have, maybe two or three times, written referee reports for such articles as (what I assume to be) favors to the journal.

On the other hand, I was only able to do that once I'd become comfortable writing reviews on subjects that I actually did know. I would have really struggled to review an article outside my wheelhouse without such prior experience, as in the case of a first referee report that the OP suggests.

[Daniel Weltman]

To add to what others have said: try to know your own personality and predilections and try to think about whether you are likely to underestimate or overestimate your expertise. I think many philosophers (especially philosophers wondering whether they ought to referee a paper) are of the temperament that they underestimate their own expertise (or overestimate everyone else's, which is the same thing). You might not be the perfect person to referee a paper, since you don't know as much about the topic as, say, the 5 top-most experts in the field. But maybe the journal has already asked those experts and they've said no, and maybe the journal has already asked 5 people who know about as much as you and they've said no too. If you say no, the journal might ask someone who knows less than you.

So, although it's important to be modest and not to referee something on a topic about which you're clueless, I think it's also important to realize that we can't just always have perfect referees for every paper, and you might be better than the next alternative.

(This is especially relevant for a paper that someone has presented at lots of conferences, etc. At that point there may be no hope for anonymity among the experts in the field, since they have all seen the paper presented, sometimes even multiple times. So the reviewer is going to have to be someone who is not one of the leading experts in the field.)

[Newly Dr]

I just want to add another point specifically on expertise worries: with all of the reviews I have been asked to do I have been able to read the paper before responding yes or no to the request. In one instance I was not sure whether I would be able to review the paper from the abstract. But after skimming the paper briefly I could see that I was able to review the paper (and that the abstract needed to be revised to better match represent the content of the paper). If you have half an hour to skim the paper and see if that makes you feel more or less confident in your ability to review it I would suggest doing so.