How often should one referee for journals?

In our most recent "how can we help you?" thread, a reader writes:

I'm at the point in my career when I'm just starting to receive fairly frequent requests to referee journal articles, and I'm finding myself really unsure of how many of these I should be doing. Would cocoon readers be willing to share their own approaches to this? How much refereeing do people do? And do people have good rules of thumb for deciding when to say yes and no to referee requests? (E.g., a certain number of reports per month, or a certain number of reports per reports one has received oneself, etc?) This is one of those mysterious parts of the profession where I just have no idea if I'm pulling my weight, doing way more than I should be doing, or roughly getting it right. Thank you, everyone!

Excellent questions! One reader submitted the following response:

At one point, I was refereeing 20 papers a year, and working as an editor for PLOSOne, handling about 6 papers a year (which meant getting referees and making a final decision on papers). Now I am a co-editor on a journal, and I no longer do PLOSOne. So I referee only about 12 papers a year, and only in a select set of journals (the best in my sub-field or very highly regarded general journals). I also review a few book proposals and manuscripts a years for various presses. Without knowing your circumstances, I think if you are doing 12 a year, then that is fine.

And another pointed out that we discussed this issue at length in another post a while back:

Your question was answered pretty extensively in this previous post: https://philosopherscocoon.typepad.com/blog/2020/09/how-often-should-one-review-for-journals.html I'm personally sympathetic to the view (expressed by several people in that discussion) that one should do at least as much refereeing work as one creates. So for everything one submits (including, of course, resubmissions of revised papers, and submitting a paper to a new journal after a rejection), one should referee two or more papers (since there will in most cases be two referees for each of your submissions).

I pretty much stick by to what I wrote in that earlier post. I try to referee at least as many papers as I submit, and think this is a good rule of thumb. But I'll add one further thing to the worries that I raised about refereeing 'too much.' The main worry that I raised in my earlier post is that if you're doing too much reviewing, that runs a particular moral hazard: namely, a relatively small number of referees such as yourself having too much power over what gets accepted or rejected at journals. I still believe that: we shouldn't want a small number of gatekeepers doing most of the refereeing. However, now that I review a lot more than I used to (I think I refereed about 20 papers last year), I have another worry: that if you review too much, you may do a worse job as a reviewer. As an author, I've sometimes received referee reports that came across to me as perfunctory and uncharitable. Other times, I received excellent reports: ones that, even if they recommend rejecting the paper, give detailed justifications as to why. As a reviewer, I think it's my duty to give the latter kinds of reports: ones that are charitable and fairly detailed, regardless of the editorial recommendation that I provide. But here's the thing: my sense is that the larger the number of referee assignments one takes on, the greater the temptation may be to do a worse job--to read papers more quickly, write reports more quickly and so on...making it more likely one may become a bad referee!

So, this is my suggestion: try to review at least as many papers as you send out, and review papers that you're well qualified to evaluate--but don't take on too many. If you do, you may end up doing authors, editors, and the profession more broadly a disservice. But these are just my thoughts. What are yours?

Comments

[G]

Please delete it if it is not allowed, since this is more like a follow-up question. I am curious about "review papers that you're well qualified to evaluate." Marcus, can you specify a bit further how you understand "being well qualified" here?

For example, if I am asked to review a paper on a topic in my AOS, and while I have a basic understanding of the topic, I have not kept up with the on-going discussions, do you think I am well qualified as a reviewer?

I usually say no to such requests. I feel like I am not well qualified as a reviewer. However, since I work at a teaching-oriented school and only have time to keep up with on-going discussions in a fairly narrow area, I have only reviewed 2-3 papers in a couple of years. I wonder if I should do more.

[Chivers Butler]

Re: G's question

Might it it be appropriate to accept with a caveat (e.g., by writing to the editor accepting the request while making explicit that one is not well-placed to comment on whether the paper is appropriately responsive to recent literature)? Provided that the editor can ensure that the other reviewer is well-placed, this shouldn't be a problem. My guess is that editors would welcome this kind of acceptance over a rejection, for it may save them considerable time.

[Julia]

I usually do about one journal paper per month, plus a bunch of papers and abstracts for 3-4 conferences per year, and a book manuscript maybe every other year. I've also done a fair amount of grant reviewing recently, and I scaled back the journal reviewing when I was doing the grants. I usually accept requests when they are either pretty interesting to me or when I am in a position to do them relatively fast. I don't usually have more than two open requests, so I can meet whatever deadline is given by the journal. I tend to write between 1 and 2 pages of comments. I want to explain the main reasons for my verdict, but I am not going to give the author detailed comments like I would for a grad student. I write the most detailed comments for R&Rs.

[William Vanderburgh]

I think the right answer, especially as a pre-tenured person, is to review only as much as does not prevent you from carrying out a robust research program of your own. Burying yourself in reviewing can seem virtuous, but it can also be a mode of procrastination and avoidance.

[Overseas Tenured]

As a rule of thumb, I try to review twice as many papers per year as the number of times I submitted something to a journal that year. The rationale is that papers usually get assigned to two referees (sometimes more; sometimes only one, or they are desk rejected), so x submissions per year generate around 2x reviews. Doing 2x reviews then more or less makes up for the amount of refereeing work one imposes on the system by submitting papers.

This is the default; but I think it's OK for pre-tenure and especially non-TT folks to do a bit less less, and tenured professors in cushy, low-teaching-load positions could certainly stand to do more. Sadly, it's often exactly the other way round.

[Shay Logan]

I generally try to give as many referee reports in a year as I expect to get. So if I expect to send out five papers and expect each to be refereed twice, I feel like I should referee at least ten papers. In practice, I go a bit higher than this---after I do the amount of refereeing I feel I *ought* to do, I then do, in addition, all the refereeing I actually *want* to do. And, invariably a few incredibly interesting papers get sent my way after my 'quota' has been filled.

[Marcus Arvan]

Hi G: sorry for the delay in responding. I'm running a workshop this week in addition to teaching, so I haven't had much spare time.

To answer your question ('can you specify a bit further how you understand "being well qualified" here?'), I don't have a simple answer. I think it's pretty much a judgment call. If I'm really up to date on a given literature, then I feel pretty confident. If, on the other hand, a paper is broadly in my AOS (ethics, political, etc.) but in a niche area that I don't really work in and haven't followed super closely, then I may be less confident.

How do I handle these cases? Again, I don't have a precise formula--but I personally find it to be very helpful when journals permit me to review the paper itself (as opposed to merely an abstract) before I make a decision. If, for example, I preview the paper and (given my background) it seems to me to have fairly obvious problems, then I may elect to review it, as I will feel well-positioned to comment on those potential problems (even if, let's say, I'm not 100% up to date with the literature). Conversely, if I preview a paper and I *don't* see any obvious problems--if it seems interesting and generally well-argued, but I'm not a super expert in the area--then it's a more difficult decision. Depending on the case, I may choose to review it (if I find the paper really interesting), or alternatively, think I should 'punt' it (as it were) by saying no and hoping the editor chooses another referee with more expertise.

Again, I think the details matter a lot here, and it's a case by case basis. I try to err on the side of, "Would I be doing right by the author--and editor--by taking this assignment on, or, based on a quick read of the paper, I am not so sure?" If the later, I may decline the invitation. I don't want to inadvertently be 'reviewer #2' by taking on an assignment that I'm not fit to evaluate... ;)

[G]

Hi Marcus, thank you so much for the response. It is really helpful!

[OP]

Dear all, I just wanted to say thanks so much for the responses (and for the link to the previous discussion...sorry for missing that earlier!). All of this has been super helpful.

[Marcus Arvan]

OP: glad we could help!