By Trevor Hedberg
One of the Cocoon’s recent posts concerned a reader inquiry about why it takes some referees longer than 2 weeks to review a paper. This is an understandable question – it’s something I wondered about when I was a graduate student – and the ensuing discussion generated an array of responses. Some argued that it wasn’t excusable to take longer than 2 weeks to review a paper; others argued that taking longer was often justified due to busy schedules and a host of other reasons. I thought it might be worth describing my own approach to refereeing papers, how it has changed as I have moved to new jobs, and why I now rarely complete a referee report within two weeks of acceptance.
When I held postdoctoral positions, I reviewed papers pretty quickly. I would read the paper during an afternoon, think about it for a few days (unless it was of such poor quality that it did not warrant much reflection), and then write up my comments during another afternoon. I also did not receive a ton of referee requests, so I typically just had one article review on my to-do list.
I have not been able to maintain that swift completion time in my current position. The biggest reason why is that the demands of my current post are significantly greater than they were during my postdocs – I teach more classes and have much greater service obligations. There are only so many hours in each week that I can reasonably devote to service without a drop in the quality of my work and an increasing risk of burnout. Some of those hours get consumed by meetings, workshops, and various forms of paperwork. I also do a fair amount of reviewing that does not involve philosophy journals. Last year, for instance, I reviewed hundreds of scholarship applications for the W.A. Franke Honors College and evaluated 52 student writing samples as part of a general education review conducted across the University of Arizona, Arizona State University, and Northern Arizona University.
When I agree to review for a journal, that referee report gets lumped into the broad class of service work I have on the agenda, and it will often not be the most pressing service work I need to complete. Peer reviewing articles is unpaid labor and not too valuable in the context of tenure and promotion. Moreover, service work in general is often lower on my priority list than teaching obligations and most of my research commitments. Thus, reviewing a paper for a journal will almost never be near the top of my priority list at the time I accept the request.
As a brief aside, some of the comments on the original post suggest that delaying completion of a review evinces a lack of enjoyment of philosophy or an improper prioritization of one’s time. I find that to be a very uncharitable interpretation of reviewer behavior. I like reading philosophy, and reviewing works-in-progress also gives me insight into what topics and recent publications people in my subfields are engaging. I certainly regard it as worth doing. But reviewing takes time, and I am expected to prioritize many other types of work above peer reviewing. Choosing to put off reviewing a paper to complete other tasks that are more centrally connected to one’s job and potential for tenure or promotion is not evidence of a character flaw. And as some commenters mentioned, editors prefer reviewers to accept and take a bit more time than to decline invitations since it is often difficult to find reviewers who are willing to accept the invitations at all.
Some folks in the comments of the original post suggested that they still never have trouble reviewing papers in 2 weeks despite having many additional responsibilities. That’s impressive and laudable, but given the variability in people’s other commitments, it is too high a standard to expect of everyone. Here are a few other factors that can affect the ability to complete a review in a timely fashion.
First, papers have variable lengths. Other things equal, reviewing a 6000-word paper takes much less time than reviewing a 12,000-word paper. It is relatively rare to get a 12,000 word paper to review because most journals have upper limits lower than that, but I have reviewed several papers that are about 10,000 words and have 2000 words worth of footnotes. (In such cases, there are almost always footnotes of such importance that they should be in the main text.)
Second, related to the point above, the number of work hours required to complete a review varies considerably. For me, the minimum time commitment to review a paper, write a useful referee report, and submit that report is about 3 hours. But the only time I complete a report that quickly is when the paper is nowhere close to ready for publication. Typically, the paper will have obvious, significant problems or (more rarely) not be a topical fit for the journal. Even in those cases, I try to provide comments that might help the author understand where the paper falls short and needs improvement. I estimate that around 40% of the papers I have reviewed fall into this category. On the rarer occasions – maybe 10% of the papers I review – where the paper is very good and only needs minor tweaks to be ready for publication, the paper usually takes a little longer to review but not much. (It mostly depends on what minor suggestions I offer and how many of them there are.)
What about the remaining 50% of papers? These take by far the longest to review. They are not so obviously flawed that I can immediately recommend a rejection on the first reading, but they still have identifiable problems significant enough that a major revisions verdict is the best I will provide. One reason these reviews take longer is that I often have to look up and read papers that the author heavily references to make an informed judgment. I certainly do not do this for every item in an author’s reference list, but if a specific source is playing a key role in their argument (or raising a key objection they are responding to), I will investigate that source if I am not already familiar with it. I also sometimes survey the literature a bit if I find the reference list to be unusually minimal or question how they are framing their project. I have been surprised by how many papers say, “No one has argued for X before, so that’s what I am going to do” and then I can find a handful of papers that have argued for X or variations of X in a simple search on PhilPapers. Authors often overstate the originality of their work. For these reasons and others, most papers in the maybe-reject / maybe-revise category receive rejection verdicts from me.
For papers that I give revise-and-resubmit verdicts, I tend to put a little more effort into clarifying my concerns and offering suggestions for changes. I do this because there is a good chance that the author will resubmit the paper to the journal and that I will be asked to review their revisions. All told, while 6 hours is probably close to the average amount of time it takes me to review a paper, some take 10 hours or more to review. (I am sure I could review these papers faster and less rigorously, but this would foreseeably result in reports that are less accurate and less helpful to authors.)
Third, I often get several review requests in quick succession. As an illustration, in one week this summer, I received requests to review 4 different papers and to review a book proposal (which includes sample chapters totaling 80,000 words). That’s an extreme case, and I simply could not accept all these requests simultaneously unless I was going to discard a host of other plans / commitments for June and July. But even under more normal circumstances, I am typically refereeing 2 or 3 papers at the same time. When I have time to review, I prioritize the article with the nearest deadline. Typically, a few weeks go by before I finish reviewing the first paper in the queue. Only then do I turn to the next one.
Even with all this considered, there are limits to how long a referee can reasonably take to review a paper. If a potential referee cannot identify a window where they can review the paper within the next 8 weeks, I think they decline the invitation and recommend a few other potential referees. Given all the factors I have mentioned, I think completing referee reports within 6 weeks of acceptance is a suitable target for most professors with moderate to high teaching loads. Completing them faster is meritorious but beyond what can be reasonably demanded of most reviewers.
IMO, refereeing is like teaching in that it will suck up as much time as you want to give it--but also, beyond a certain point, all the extra effort you put in doesn't really move the dial very much (for editors or authors) any more.
I try to stop before that point of diminishing returns.
Posted by: Michel | 07/08/2024 at 07:51 PM