Improving our current peer-review process

As I'm sure most of the blog's readers know, I've advocated for dramatic changes to our peer-review system--ones I've argued would be likely to benefits authors, reviewers, editors, and the profession. While some surprising changes in policies by European funding agencies might push things in the direction I've advocated (though that remains to be seen), a fair number of readers have asked whether it might make more sense--at least in the short-term--to examine less radical ways we might improve our current system of peer-review.

A few comments the last several days got me thinking more about this. First, Angry Anon wrote:

I send only to generalist journals that I know are quick (1-2 months), and after that only to specialist journals that I know are quick (and are usually quicker).

Here's the real question though - and maybe we could start a separate post for answering this question - why are you, as a referee, taking longer than 3 weeks to review a paper? It is unreasonable and immoral to take longer than that. (Yes, you are a bad person if you take longer than 3-4 weeks to review a paper.) I have never taken longer than a week to review a paper. That it takes anyone longer is ridiculous (save for extremely long papers that might take 3 weeks or so). What ever happened to treating others how we would like to be treated? Does anyone like having their paper getting hung up for 3 months minimum? If you aren't going to review it in 3 weeks, don't accept the invitation to review.

And what about those grossly immoral referees who take longer than 3 months? I've had a paper take over a year only to get an R and R. (And no, I didn't revise and resubmit it. I'm not going to send it back to some jerk who is going to take that long to review it.) What was this jerk doing that whole time? Did it take less time to review after waiting 12 months?

On a similar note, Frustrated reviewer wrote:

What takes more time, finding reviewers or waiting for them to hand in their report? Two months ago I reviewed a paper which is, as of today, "awaiting reviewer assignment" according to their manuscript management website. Apparently, they can't find a second reviewer. It's frustrating not just for the author. Although I recommended some revisions, I find the paper thought-provoking and would love to see it succeed, but it seems that there is nothing I can do to help get this paper out into the open. Maybe I should inquire with the editors why they let this promising paper linger in review hell or are so lazy? Has anyone of you ever done this?

While I don't entirely agree with Angry Anon's standards for reviewers. While I do think there are reviewers who take too long (more on this momentarily), I don't think Angry's suggested three-week turnaround time is doable for everyone. I've reviewed dozens of papers, and I've never missed a single deadline. However, during the academic year, when I have a full teaching load and a long list of other duties, three weeks may not be feasible. I need time to read the paper, think it over, write a detailed report, and then set the report aside for a bit and return to it later to make sure I am being fair to the author. During a hectic semester, I think this can require something like two months--especially if (like me) you sometimes take on multiple referee duties, get unexpected R&R's of your own that you have to work on, and so on.

For my part, I think a two month standard is reasonable. However, one problem--at least in my experience--is that this reasonable standard (not to mention a three month standard) is routinely not met. As you can see here, quite a few journals in our discipline have turnaround averages well upwards of four months--and even those with 2-3 month average turnaround times have plenty of deviations upwards of that average. For instance, like Angry, I tend to target journals that claim good turnaround times. However, the last four or five papers I've submitted have all taken longer than four months. Given that that's 1/3rd of a year, it's very frustrating (particularly when sometimes, after all that time, the review is little more than a short paragraph!). Why does this happen, and what can be done about it? In my experience (having emailed editors to find out what's going on), there appear to be two common explanations of long turnaround times:

1. Difficulty finding reviewers
2. Difficulty getting all the reviewers to get their reviews in on time.

However, when you look at journal statistics, some journals have far less variance in turnaround times than others. This suggests that some journals have good methods for ensuring that they both (A) get reviewers in a timely fashion, and (B) get their reviewers to get their reviews in on time. It might be good to hear from editors, both those who struggle with these issues as well as those who have found good methods for handling these issues. Any editors willing to chime in?

Although I've never been an editor, allow me to briefly speculate on a few things. I've noticed that some journals ask authors to recommend reviewers. While in principle that could lead to abuse (viz. authors nominating their friends), this seems to me like it might be helpful in securing reviewers, and that abuse could be protected against--for instance, by the journal informing authors not to do this, and instructing reviewers to recuse themselves if they have any idea who the author is. By a similar token, I've noticed that some journals ask reviewers to specify other reviewers. This may be helpful too. What about turnaround times? Here, I suspect one thing that can help is setting shorter deadlines. For instance, as a reviewer I've noticed that some journals state they expect a review in three months, whereas others say 6-8 weeks. Given how easily people can procrastinate without clear deadlines, I suspect it's probably better for journals to set shorter deadlines. Finally, I think it might probably help if there were 'carrots and sticks' for referees--that is, positive incentives for getting reviews done on time, and negative incentives for not.

I'd also like to suggest that editors be more proactive about keeping authors in the loop. Whenever one of my papers has been under review a long time, I stress over whether to contact the editor--as I don't want to rub them or reviewers the wrong way by 'bugging them' (though I do want to know what's going on with the paper). Instead of having to bug editors, I'd suggest that perhaps editors proactively contact authors when their stated (e.g. 6-8 week deadline) has passed, letting the author know why, such as "We had trouble getting a second reviewer, and expect their review by Date X", or even giving the author some information about the reviews that have been received so far, "We received one review and are waiting on the second. The first review was mildly negative. Bearing this in mind--and bearing in mind that we normally require two positive reviews for acceptance--would you like to keep the paper under review." In addition to working to improve turnaround times (and consistency of them), I would think something like this approach might make the peer-review process a better experience.

What do you all think?

Comments

[JR]

My experience is that a reason for why peer review take so long is because of the personality of the referees. I know people in my field that gets nothing done until very close to the deadline. They make conference presentations during the flight to the conference destination. They do not respond emails on time but after several queries. They (and many admit this too), do the peer review day or two before the deadline. If the journal gives referees two weeks to complete the review, they do the review on the last day before the deadline. If the editors give them 4 months to do the review they do the review on the last day before the deadline. In my personal opinion, these people are annoying and difficult to work with. Luckily there are also people who always reply to emails first thing, do the reviews on the same day when editors ask for them to review the paper etc. We should have more people like the latter and less like the former in academia.

[Amanda]

I think all of these suggestions are worth trying, but in the end won't make much of a change. The exception might be shorter deadlines, more on that below. As for reminders, those who are motivated by reminders are the same people who already don't take very long to review, and those who do take a long time will not be motivate by reminders. Again my solution which won't happen: charge for submissions (a sliding scale based on salary) and then pay reviewers.

I also think there should be a four week deadline. I find it hard to see a reason why someone cannot finish it in four weeks, or maybe 6 if they ask for an extension. Marcus you can cut out the needing time away, I'm sure your reviews are already better than most. Anyway, time is fungible for the most part, and reviewing takes a certain block of time that will be the same block whether it is done 2 or 10 weeks from the day someone got the review. In the rare circumstances when someone really can't meet 4-6 weeks then they should decline. Of course, people violate deadlines, but I tend to think that most people will only violate them a certain amount, so shorter deadlines will lead to overall shorter reviews, even if they far exceed the deadline. And like JR said, some people do everything last minute, so in this case a shorter deadline would help.

[Cameron]

This is a bit off topic, but is there a reason journals don't have staff reviewers? This seems to be another area where in any other career there would be people paid to do this work rather than a reliance on free labor.

I suppose having staff reviewers might be difficult for a generalist journal, but surely something like Ethics could have a small team that handles most of the reviews. They could then contract out review work if no one on staff had the required expertise. Even contract work would be better than expecting people to do it for free.

[Marcus Arvan]

Hi Cameron: I have worries about the notion of staff reviewers. I worry it would place the peer review process in the hands of a small number of people, giving their perspective an outsized influence over what gets published. I’m inclined to think widely distributed, less centralized decision making is better for protecting against bias and “journal capture”.

[Cameron]

Hi Marcus: That is certainly an important worry, but I'm not sure it outweighs the problems of the current system. Since each journal would have different reviewers, you could just send your paper to another journal if it gets rejected. If the bias is so widespread it carries between all or even most journals, then it's probably already a problem for the existing system.

Personally, I'd rather have a journal turn down my paper in a week or two (which I'm assuming would be a reasonable time-frame for dedicated reviewers), with a clear explanation, than wait for months only to have the paper rejected without explanation. Even if the clear explanation is "this is not philosophy", at least I'd know not to send that journal any more of my work.