How do you decide when to review for journals?

In our new "how can we help you?" thread, a reader writes:

I have been post phd for a few years now. At first, I thought it good to get some referee work under my belt. Now, I am getting a request to review at least once a month if not more. Today, I got two requests 15 minutes apart. My question is, how do people approach the decision whether to accept or reject a request to review? What are the considerations people take into account (besides fit to review)?

Great questions, and I'm very curious to hear what people think! As mentioned in the new thread, to make things more manageable on my end, I've asked readers not to respond in the comments section there--and so there are several comments that I have not approved. However, although I did not post them, I will still try to quote them anyway in new posts when relevant. So, on that note, Bryce responded:

A colleague once gave me this advice about reviewing: review as many papers per year as you submit. So maybe 1-3? The more you review, the more you will be asked for review. Just say no.

This is a rule of thumb that I have commonly heard too, and it seems to me a good one. As professionals, we should all do our part to contribute to what are, effectively, 'public goods' in the discipline. If you submit a lot of papers for others to review, then you should review a lot of papers. If, on the other hand, you submit few papers, then feel free to review more rarely. What does seem clearly wrong to me is to submit lots of papers and rarely review when asked. That's just free-riding. But let me add a few further thoughts.

One thing that often affects whether I choose to review a paper is the opportunity to preview the paper before agreeing to review it. Many journals (though not all) do this, and I think it is a great practice. On the one hand, if upon skimming the paper, it seems very promising to me then I may accept the invitation because I have a sense that the paper should probably be accepted (or at least given an R&R) and I wouldn't want to have the paper face a less-sympathetic reviewer (since, as we all know, some reviewers can be ridiculously unsympathetic). Conversely, I may choose to review a paper if I skim it and think it has very serious problems that would be bad for no one to catch prior to publication. A simpler way to put this is: it matters to me whether, upon skimming the paper, I think that I would be a particularly good reviewer for it, given my expertise and what it does. 

Although I think the 'review 1 paper for every 1 paper you submit' is a good rule of thumb--at least in terms of doing one's fair share--I don't think there's anything wrong with doing a bit more than that. I think I reviewed 16 papers last year, despite submitting nowhere near that much. Basically, I figure that the journal system is so swamped that if people like me (who have stable jobs post-tenure) can help a bit more, then we should. However, on that note, I want to raise a worry about reviewing too much that I don't recall seeing discussed. Every once in a while on social media, I've come across (usually senior, well-established) figures who report reviewing ridiculous numbers of papers each year (like 30-50+). I'm not entirely sure why they do it, though I imagine they think they are doing good. But, for my part, I have the following concerns about it: it suggests that a relatively small number of people may be serving disproportionately as gatekeepers for what gets published. Why is this (potentially) problematic? Well, because while expertise matters, everyone has their own biases, axes to grind, entrenched views and intuitions, and so on. The danger, I see it, is that if some people review vastly more than others, then the publishing system as a whole (or at particular journals) may in essence become 'held hostage' to the views and biases of a select few. Epistemic humility, I think, suggests we should want peer-review to avoid that. So, I say, review a fair amount--at least as many papers as you submit. But also don't over-review, taking yourself to be a member of a select few who simply 'must have your say' in the peer-review process as much as possible. 

But these are just my thoughts. What are yours? To repeat the OP's questions: "how do people approach the decision whether to accept or reject a request to review? What are the considerations people take into account (besides fit to review)?"

Comments

[Random R1 Professor]

I'm fairly senior now and get asked to referee quite a lot -- though I'm also in a small subfield, so "a lot" isn't unduly burdensome. I very rarely say no. (In a typical year I referee four or five times as many papers as I submit, but that says more about how little I submit.) Occasionally I'll see that a paper approaches the subject in a way for which I have no patience, and I don't think it's right to review papers for which I can't muster even minimal sympathy. Otherwise I pretty much always say yes.

I can't imagine doing a good job with 30+ papers a year, and I certainly agree that the gate-keeping worry is a serious one.

[Overseas TT]

I'll chime in. First of all, for what it's worth, reviewing twice the amount of what you submit might be a better approximation. A submission is looked at by 2 referees in average, so a submission requires around two referee reports. This would justify that we referee two papers for every submission, not one.

Having said this, I get way more requests than the number of papers I should referee per year even by this more demanding standard, so I rule them out on the basis of a few rules of thumb (I'm putting aside obvious cases, like when I identify the author as a good friend).

1. Some journals ask me to referee much more than others. If I already refereed a few papers for them recently, I feel much less pressure to agree yet again.

2. I absolutely never, ever referee a paper if I already recommended its rejection at another journal. I hold myself to this rule religiously. So should you. No matter how terrible I thought a paper was, I strongly believe that it would be an abuse of power for me to give it the thumbs down at two different journals. I consider this something like an equivalent of ballot stuffing. I cannot begin to explain how infuriated I am when I'm on the receiving end of a copy-pasted referee report. It's the height of arrogance, and I'm amazed it's not obvious to everyone that you just don't do it.

2. I don't referee papers on topics on which I'm competent but which I find so boring that I can barely maintain focus while reading on it.

3. A bit more controversially, I'm somewhat less inclined to referee for journals where I feel like the editor is trying to manipulate the outcome. Sometimes I felt like editors didn't simply disagree with my judgment (which is their right) but were softly pressuring me to turn a rejection into an R&R.

4. Finally, I don't referee for editors who were in the past rude and disrespectful to me. This isn't common but it happened, I'm sad to say.

5. I'm not less likely to referee papers whose starting assumptions I reject. I don't feel particularly attached to the views I hold in philosophy, including the ones I defended in print, so this doesn't make it likelier that I'd reject something.

[Tim]

I go by the at least 2 reviews per my own submission.

[Tom2]

I basically review whenever I'm asked. I enjoy it. It's one of the more gratifying parts of being a philosopher and it helps me keep on top of the literature. With R&Rs it comes to about 2-3x / month.

Once I was reviewing an article for journal X and got a request from a different editor at X to review another paper. That one I turned down.

I agree with Marcus that it's important to be able to skim a whole paper before agreeing to review. Sometimes I can tell from an abstract that I'm competent to review; other times I can't.

One thing that has struck me as a reviewer is just how much competent work is out there. Even when I reject things, the papers are never disasters. They just have problems big enough to warrant rejection. Almost always there's an interesting and worthwhile nugget which can, with work, be made publishable.

I'm quite down on the profession these days, but I still love publishing philosophy--my own and, in a small and indirect way, by working as a reviewer. I know people complain a lot about the journals, but I think most of them do a great job and are one of the bright spots of the philosophy profession.

[other opinion]

Let me add a controversial opinion: I don't think you have any obligation whatsoever to review for journals and on the margin I think you shouldn't do it. It would be better for all of us if referees were paid. Why should we supply this unpaid labor while publishing houses make profits? It is not in our interest. We should not fool ourselves into thinking that we owe this labor.

Now others might add that refereeing is contributing to a public good and therefore one should contribute. I don't buy that argument. First of all, journal publications are by definition not a public good, because they are an exclusive good -- you have to pay for access to them. At best this argument works for open-access publications.

Second, even if we think that referees contribute to something good (rather than a broken and exploitative peer review system, as I tend to see it), that doesn't mean that they shouldn't be compensated for it. The people regularly cleaning up national parks should be compensated for it, even though they contribute a public good. The same would be true for us.

Third, you might argue that we are obligated to contribute referee reports because you also rely on other people doing so. I don't find that plausible when it comes to unpaid labor. Assume that I rely on textiles that are produced through unpaid labor, because those are pretty much the only ones available. Do I therefore also have an obligation to provide unpaid labor for producing such textiles? That seems ridiculous! It would just be better if the people producing the textiles would get paid -- and if you are a person who can produce textiles you should go on strike to make that more likely to happen. By analogy, you should not review for journals.

I also think we should get paid for our publications, by the way. I understand that prior to tenure we are incentivised to publish without direct payment, but why would we continue to give our valuable research away for free to commercial publishing houses once we have tenure? Now one might make a special case for open access, non-profit publishing, but that is still not the standard.

[On a related point...]

On a related point (Marcus - please feel free to remove if you think it not related enough), I'm interested in what people think of this: https://medium.com/@jamesheathers/the-450-movement-1f86132a29bd
The idea of asking for payment for peer reviews is not new, but this puts it in a new (for me at least) way. The author also says that they would only send this to journals that are run by large for-profit companies, so open access/society journals could be exempted from any requests for payments.
My thoughts are that something has to change, though whether this is the right route is another matter.

[Original asker]

Thanks for the responses. I want to focus the discussion away from questions of duty and obligations towards more prudential realms.

For instance, one of the papers I was asked to review is by an editor at a journal I know personally. I plan to submit a paper to that journal in the next few months. I recently (2 months ago) reviewed a paper for that journal. The paper is in my AOC, but not my AOS. Prudentially, should I just go ahead and do it?

Another paper I was asked to review is in my AOS in a top journal I have never reviewed for before. But it seems like more than I would like to do at the moment (given a few other reviews just completed). Nevertheless, I would eventually like to submit a paper to that journal.

I guess, my question is, given that editors don't know the background, when they get a refusal to review, should I worry about pissing them off? I am sure most are decent and not vindictive, but some are, and some are unconsciously more positively disposed towards those who 'played nice'. How much should I worry about being a 'team player' in editors' eyes?

[Original asker]

I will just add that one reason I don't want to discuss duties and obligations is that I am pretty sure I am fulfilling those (e.g. over 2 reviews per paper submitted).

[Peter Furlong]

My own view is that by the end of my career I should have refereed at least twice as many times as I submitted (since each submission tends to require two referees). In the beginning, obviously, I was submitting much more than I was refereeing, since no editors knew me or my work well enough to ask me to referee. These days I am getting too many referee requests to handle, and more than twice as many requests as submissions. (This is partially because I have spent most of my time the past few years working on books rather than articles, so my submission numbers have gone way down. This past year, though, the number of referee requests I receive has gone way up, too.) My policy on looking at a paper I have refereed elsewhere is that I refuse to do so unless the paper has been dramatically rewritten, and even in cases of dramatic revision, I let the editor know the situation.

[other opinion]

@Original asker

The prudential question is harder to answer because none of us has good data. I presume that none of us randomly selected a bunch of journal for which they rejected review requests and a bunch of journals for which they didn't.

I am personally not too worried about editors being vindicative. My hunch is that they have too many rejections and requests to deal with that they don't keep even a mental list of the baddies. But that is just a personal guess and comes without warranty. It also depends on how risk-averse you are. If you are more risk-averse the calculation might come out differently.

When the journal practices triple-blind peer review than the worry should be zero (assuming the process is not compromised). So if you want to reduce your refereeing load in the future (probably not applicable for the specific cases you describe), my prudential suggestion would be to reject requests from journals practicing this type of peer review.

[Overseas TT]

Just a quick comment: I agree in principle with "other opinion" that we shouldn't referee for journals for free, at least not for journals published by for-profit presses. But I think this raises tricky questions about collective action. Perhaps if enough of us refuse, refereeing at those journals will someday be compensated. But in the short-to-medium run, the only people whose lives we are making more difficult are the editor (who is normally blameless) and the author, who often struggles to secure a job, or tenure, and may be in need of a quick verdict. Still, I do agree with the policy in principle - I'm just skeptical that individual-level action is an even remotely efficient means of implementing it.

[Evan]

The argument for getting paid to referee is interesting. I suspect that if journals started paying for referees, they’ll be incentivized to be highly selective in *who* they want to referee or edit for them. I think this is because there can be free-riders who want money and may end up doing poor or biased reviews because they’re primarily motivated to make money.

Another unintended consequence that may arise is that those who really want to referee may not have a lot of opportunities because new checks and balances on refereeing might be created to prevent them from doing so.

I think the rationale is that by not paying referees, they’re motivated not by money, but by a commitment to high-quality or competent inquiry. It’s difficult to assess whether or not a referee is reviewing in good-faith as opposed to doing a poor review in order to make (easy) money in that scenario.

Then again, it’s probably not going to be ”easy money” since journals may require referees to meet other criteria in order to qualify to referee. As a result, this may increase the gate-keeping phenomenon.

Of course, one could argue that it wouldn’t make any difference in the quality of refereeing or articles published if journals started paying since there are poor-quality and problematic refereeing practices that currently exist.

One could respond to this by arguing that payment might *increase* the number of poor-quality refereeing or articles since people will probably just do the bare minimum of what is expected of them in order to get paid.

It really depends on who actually likes refereeing and has a sufficient level of epistemic humility to do so. Just something to think about.