How much time do editors and referees spend on submissions?

In our newest "how can we help you?" thread, a reader asks:

How much time do editors/referees spend on average reading submissions before making a decision? Do they read them more than once?

Interesting questions. It would be great to hear from readers (both editors and reviewers), but I suspect there is huge variance. As an another reader writes in the following submitted reply:

I suspect the average would be misleading -- a distribution would be much more helpful. Such info, while likely not feasibly accessible, would be interesting to see (probably there's a *huge* variety).

In my own case as a reviewer, how long I take on a paper depends on many things: how long and complex the paper is, how good I think it is, how certain I initially am about the recommendation I plan to make to the editors (sometimes I know that I want to recommend reject, accept, R&R-major revisions, etc.; other times I waffle), how much work I think a paper needs if I recommend an R&R (as in these cases I write extremely detailed reports specifying exactly what issues need to be addressed for me to recommend acceptance), and so on. Then, of course, there is my personal and professional schedule. Some times of the year are busier than others with teaching, family obligations, service obligations at my university, research (sometimes I have revise-and-resubmits and other research deadlines to meet too!), and sometimes these things can be planned for but other times not. All of these things and more result in huge variance in how long it takes me to get a referee report done. Sometimes I get reports done well in advance of the journal's deadline (in fact, sometimes after just a few days or weeks), but sometimes it takes considerably longer. In any case, I have a policy of meeting journal deadlines, and I'd never not complete a report on time (or at most a day or two late).

But this is just my experience. What are yours? It would be great to hear from reviewers and editors! 

Comments

[Philosopher]

It seems like there are two questions being asked here: 1) How long do referees/editors spend *actually on* a paper, i.e. reading it, thinking about it, writing a report, and 2) How long do they spend *sitting on* a paper, i.e. how much time generally passes between the point at which they get the paper and the point at which they make a report.

For 2, I also tend to get papers in within the time frame, but over time I have moved towards submitting them within the week before the deadline hits. Unfortunately it seems that when I do reports right away, I end up getting asked to do a lot more reports, and since women already do a disproportionate amount of service in the profession, I don't want to invite more. I guess I could just say no to invitations more, but again given how women tend to be judged by others, I also don't want to develop a reputation among the editors of the journals I want to publish in of being someone who says no.

For 1), I usually read the paper through once, making comments as I go on the argument. Then I read it through again more quickly thinking about a wholistic recommendation. Then I write up a report, which usually ends up being about 2 single spaced pages. I'd say this usually takes me about one full morning or afternoon - so, 3-4 hours.

[anon]

I should be better at logging hours spent on this than I am. But basically, the process usually takes a week for me. I read it once and sit on it a bit. Then I go back and read it again, then I start writing up my comments. As I do so I'm re-reading various specific parts of the text that I am engaging with in my comments as I go along.

[history of philosophy person]

Similarly to anon (1/25/2023 at 10:03 AM) above, I will always do at least a first reading without taking notes followed by a reflection period of at least a day, and then a second reading with making notes. If I'm sure of my decision recommendation, which I often am, that can be enough.

But I've also read papers as many as six or seven times when I've been on the fence and/or confused. I don't necessarily recommend this, as I think (?) it reflects my own insecurities more than good professional practice. But as a data point, this is what I sometimes need.

I always, always return the report quickly and well before the deadline. I think the longest that I've ever taken to return a report after accepting an invitation is about two or three weeks. My sense is that others take a lot longer.

[referee]

I usually do the refereeing work on a paper over two days - one day, I read the paper. The next day, I write the report. I return to the paper only as I find myself unable to express the author's argument. When I read the paper I make notes, and these are the key things I flag in the referee's report.
I do not agree to referee a paper unless I can agree to do it within a week of the invitation.
I referee about 12 papers a year; at one point I was refereeing about 20 a year; early in my career I would referee maybe 2 a year. I am asked to referee between 20 and 30 papers a year. I was on the editorial board for a non-philosophy journal, and I was handling about 6-10 papers every year. I was the one who made the final decision on the paper, based on the referees' reports. I read the paper and the reports, and prepared my letter on that basis. That work was usually spread over 3-5 days.

[Combination Editor and Reviewer Remix]

Editorial practice varies significantly so I don't think there are good general answers to the question. I also read very differently as an editor versus as a reviewer.

As an editor handling the desk rejection vs external review stage, my goal is not to figure out whether the paper is "good" or not, but whether it fits with the journal's scope (which, hopefully, has been transparently stated on the website and the submission page). For example, if the journal does not publish replies, then it is pretty quick to figure out that a reply manuscript doesn't fit. Other cases are more time-consuming, but still not as much as actually reviewing.

As a reviewer, I would typically read once to get a sense, and then again to make in-line comments (mostly for myself), and then look it over to write the referee report. That probably takes me an afternoon to a couple of days.

[in general]

This might be a question Combo Editor can answer; if not, maybe someone else can.

If the journal is a generalist journal, then presumably quite a lot falls under the journal's scope. In that case, I'm curious how the desk rejection phase vs external review phase is handled. What are the quick & dirty techniques used to desk reject at this stage at generalist journals assuming, again, that it is not hard for the paper to appear to fall under the journal's scope. Here are some rules I imagine being deployed: 'Intrinsic interestingness'; 'potential contribution clearly visible'; 'likely easy to find referees'. What else?

[associate editor]

As associate editor at a well-ranked rather generalist journal, I only get submissions that fall into my domain and that have been preselected by the editor-in-chief. I read through the paper to decide i) whether it fits well into the journal (e.g., the paper is interesting enough to a large and diverse philosophy audience), and ii) whether the article is good enough so that rejection is unlikely. If it is clear that there are too many issues for at least two R&Rs (or even better, two minor revision verdicts), then I usually do not send it out (which means that I have to justify desk-rejection to the author). I would say that I spend quite a bit of time on the papers, read them in a concentrated way, make a reviewer list (as one has to ask many reviewers before someone accepts to review), etc.

Once I get verdicts back from reviewers, it depends what and how they are. Ultimately, I decide, not the reviewers, however, if at least one reviewer is quite negative about the paper (equal reject), then it is hard to justify R&R, as the paper will usually be sent back to the same reviewers (except if the reviews are badly done, incompetent, etc.). Usually, I spend quite some time to decide, to think about whether I have additional commentaries to the author, I emphazise which comments I deem important to be addressed, etc.
However, I would say that probably every editor handles this differently, some are probably much faster than me...

[Occasional Referee]

My process is pretty similar to those described above. First I just read the paper, taking minimal notes, just trying to understand the main lines of the argument. Then I try to summarize the argument, going back to the paper if I need to. Usually, I need to, because when I try to summarize the argument, there are gaps. If I can find how the paper fills in these gaps, great! If not, then that's a comment for my report. I try to be helpful to the author in my report, even (especially?) if I recommend a "reject" verdict, so I typically end up writing around 2000 words of comments. All told, I probably spend about 8 hours on average per paper that I review, but that varies considerably with the complexity and quality of the paper.

[F]

It varies hugely. A 5-page paper for Analysis and a 60-page logic paper are two very different beasts...

[CW]

I skim the paper to get a sense for what it's trying to do, then read carefully to fill in the details. I then set it aside for a few days, maybe jotting a note or two as things occur to me.

When I sit down to write the review, I'll usually want around 3 hours. I start by summarizing the argument, then list big and small issues in separate sections of the review. I'll write two single-spaced pages for most reviews.

My sense for whether it should be published can come at any time, sometimes on first read, sometimes only later as I'm detailing what I think to be larger concerns or questions.

[UK Postdoc]

I am surprised by how long people's referee reports are! Mine are usually closer to a few paragraphs than a few pages, especially if the verdict is 'reject' - in which case I'll just try to summarise the main flaws. I might go into a bit more detail if I recommend an R&R, because in that case I want to pick up on some smaller details.

What is people's motivation for their report length? I suppose that for the editor, longer reports are not necessarily more helpful. I can see why they are for authors, so are most referees just nice people who want to help authors in their future endeavours?

[anonymous editor]

I'm an associate editor of a good generalist journal, where I handle about 25 submissions (in my field) per year.

At the very least, I read each paper very carefully once, taking notes along the way. Usually a single careful reading is sufficient for me to decide whether to desk reject the paper or send it to referees. This takes about an hour or two on average, depending on the length and technical detail of the paper. I try to complete this within a week of receiving the paper.

I spend more time, and certainly more effort, tracking down qualified referees. Many referee invitations are simply ignored, and many are declined without suggesting alternative referees. Sometimes there are legitimate reasons not to suggest alternatives--say, to avoid conflicts of interest--but I suspect a lot of people don't offer suggestions simply because they're busy. (But note: A subfield expert should be able to offer good suggestions in a minute or two, whereas it might take an editor thirty minutes of browsing philpapers and skimming relevant work to find the same people. So if you are in the position to recommend referees, editors really appreciate it!)

Once I've received referee reports, I try to issue a decision within a few days. When the referees are in agreement and the reports are well crafted, I may not read the paper again before arriving at a decision. However, I always do so when the decision borders on R&R or better (for the author). Regardless of the decision, I always write a brief note to the author explaining my decision.

As a result, my overall time commitment varies significantly by paper. Some desk rejections take less than an hour, but I might spend over 10 hours on a difficult paper that receives multiple R&Rs.

[Combination Editor and Reviewer Remix]

@ in general

Again, every journal is different, and ideally each would make its criteria as transparent as possible. For example, Philosophers' Imprint, a generalist journal, says on its about page:

"Submissions to the Imprint are refereed anonymously and selected for publication on the basis of their estimated long-term significance. The Imprint publishes articles that engage directly with a philosophical issue or historical figure. We do not publish "interventions" in contemporary debates or commentaries on contemporary figures. Although there is no page limit on submissions, the Editors value economy of expression and do not plan to publish book-length works."

Obviously, this is still going to require many judgment calls from editors, but it does give some non-topic-related criteria.

[John Collins]

I'd estimate I spend on average 4 to 6 hours reading, thinking, and writing the report.

[E]

Perhaps this is bad practice, but it often depends on the recommendation I will give. If the paper is not publishable, I usually don't spend a huge amount of time. I don't see it as my job to explain precisely how to make the paper publishable, so I simply explain to the editor why I recommend rejection. These reports are usually short - one or two paragraphs. If you add in reading time, I guess this takes about 2 hours usually.

When we move to R&R recommendations and acceptance, things inevitably take much longer. I might even read the paper two or three times (or more) and try to write more extensive comments. In these cases, I guess anything from 4 hours to 8 hours.

[Julia]

I can usually finish a refereeing assignment on a domestic plane flight.