The Philosophers' Cocoon

How to incentivize better author submissions?

In my past two entries in this series--as well as in other past discussions of peer-review--I have tended to focus on how editorial practices and reviewer behavior might be improved: to better ensure quicker and more reliable turnaround times, as well as better reviewer comments. However, it increasingly appears to me that this is to focus on only one part of the problem.

As David Velleman noted here--and others have noted as well (see below)--another central reason why peer-review has the problems is does is the behavior of authors. Journals are overrun with far too many submissions, many of which seem--to editors and reviewers--as though they should have never been submitted in the first place. This bogs down the entire system, making reviewers not only more difficult to find but also increasing their workload (as they have more papers to review). This in turn may be one significant cause of 'poor reviewer behavior', both in terms of long turnaround times and the quality of their reports. Indeed, there seems to be significant anecdotal evidence for this.

First, respondents to our poll on peer-review generally agreed with the idea that too many half-baked papers are sent out for review:

How to incentivize better peer-review reports?

A while back, I ran and reported the results of an informal survey on what readers think works well and not so well in peer-review. Although the results should be taken with a grain of salt, a number of trends emerged. One was that a vast majority (93.67%) of respondents agreed that 'journal turnaround times are too long and inconsistent.' In a follow-up post, I presented a possible proposal for how to address that issue: a central editorial system (perhaps at Philpapers) for journal editors to use that would give each reviewer a reviewer score, based at least in part on how quickly the reviewer responds to review requests and gets their reviews in. Whenever an editor gets a new paper, the system would then present the editor with the best reviewer matches--the final wrinkle being that the system would be designed to match papers with reviewers whose overall reviewer score is similar to the author's reviewer score. In other words, the system would match authors who are slow reviewers with other slow-reviewers, and authors who are fast reviewers with other fast reviewers.

Although this is obviously a fanciful system, at least at the present time (given that there is no such central system), it seems to me one that might be technologically feasible and work to everyone's advantage: authors, reviewers, and editors (particularly editors, as such a system would vastly simplify their task in finding reviewers!). By matching authors with 'reviewers like them', the system would plausibly incentivize many people to be better, faster reviewers--as there would be some incentive to be a better reviewer (namely, the incentive of getting matched with better reviewers as an author!). I also wonder whether this system might address a second set of issues that respondents in our poll cared about: quality of referee reports. As you can see below, many respondents in our poll thought that too many reviewer reports are incompetently, biased or needlessly aggressive:

Survey results on peer-review

Several days ago, I put up this informal survey to gauge readers' opinions about various features of peer review. I am going to leave the survey up so that people can still take it. But here are the results so far (n = 91), organized by what I will call the good, the ambivalent, and the bad.

As this is only an informal survey, the results should of course be taken with a grain of salt (and then some). Still, I found the results interesting and will be curious to hear readers' reactions.

The Good

64.1% of respondents agreed that anonymized review works well on balance, giving authors a fair chance (whereas only 23.07% disagreed).

An easier poll on peer-review

[Update: survey fixed!]

Following feedback on the poll on peer-review I rolled out the other day, I've tried to design an easier poll. HERE is a new survey that I think readers should have a much easier time with. Although it has 20 items, the items are so simple that I think it should take readers just a couple of minutes to click through. Hopefully we'll get some better data out of this one--and, although it's only an informal poll, my hope is that its results will give us some good material to discuss moving forward!

Reader poll: best & worst parts of peer-review

Based on readers' answers to my query on what works and what doesn't in peer-review, I've put together THIS POLL (poll suspended for revisions - see comments) for readers to rank features of the peer-review process in philosophy from best to worst. Although I'm by no means an expert at putting polls together, my hope is the results will give at least some rough indication of readers' attitudes. We'll see how it goes--and I plan to host discussion threads on specific issues moving forward--but I hope some of you take part!

Peer-review: what works and what doesn't?

Several months from now, I'm scheduled to take part in a discussion on our profession's peer-review practices: what works, what doesn't, and what might be improved. To prepare for the session, I'd like to run a new series here at the Cocoon soliciting readers' opinions on our peer-review practices and norms, and what (if anything) readers think might improve the process.

In today's post, I'd like to ask readers to weigh in on two initial questions:

1. What do you think currently works well in the peer-review process?
2. What do you think the biggest problems are with the peer-review process?

After hearing what readers think, my plan is to run a few informal polls, so as to gauge readers' overall satisfaction with the peer-review process, desire for change, and priorities if change is pursued. Then, after those polls, I hope to have a series of posts discussing some of specific issues in more detail, including more detailed debates about whether the peer-review process should change at all, if so how, and so on. 

So please do considering weighing in on questions (1) and (2). The more readers we hear from, the better discussion we are likely to have in the series overall!