Peer-review: keep, change, or abolish?

Darrell Rowbottom (Lingnan University) has drawn my attention to a new paper of his forthcoming in British Journal for Philosophy of Science in which he defends pre-publication peer-review against a recent argument by Heesen and Bright for abolishing the practice altogether. Readers may recall that over the years, I've tried to stake out a middle ground in this debate: arguing for substantial changes to peer-review, but not its abolition. In brief, my own position has been that philosophy should probably transition toward the publication model practiced in math and physics, wherein it is standard practice to:

1. Upload draft papers to the ArXiv (a preprint server similar to the PhilArchive).
2. Share, discuss, and critique unpublished drafts openly (a kind of crowdsourced peer-review that weeds out bad papers and improves good papers before journal submission).
3. Submit papers for peer-review at journals only after steps 1 and 2.

Although this model to a large extent undermines anonymous review (as people then know who has posted which papers to the ArXiv), my own sense is that the system works well and might help resolve a number of problems with our current peer-review system (including but not limited to: too many half-baked papers being submitted to journals, journal editors and referees being overwhelmed by submissions, etc.). So, then, we really have three options for peer-review in philosophy:

• Option 1 - keep the status quo: stick with the anonymized peer-review system we have.
• Option 2 - abolition: abolish pre-publication peer-review at journals altogether.  
• Option 3 - reform: keep peer-review at journals, but adopt a practice of posting papers on online servers first so that they can be openly vetted (e.g. evaluated, criticized, etc.) before journal submission.

Which option do readers most support, and why - especially, let's say, after reading the papers by Rowbottom, Heesen and Bright? For some additional food for thought, readers may want to check out this story shared over at Daily Nous  by Nathan Salmón (UC Santa Barbara).

Comments

[Brad]

Marcus
I think it is important to note that Remco and Liam are talking about science - so it is really open whether such a practice should be extended to philosophy, and whether they think it should. Part of the problem with philosophy is that we really would be swamped with a lot of low grade papers. Assume that there is only a 10% acceptance rate in philosophy. So now we would be confronted with 10 times the papers - and many that our deem to be not worth publishing. That is not the situation in physics, for example. The science generally have higher acceptance rates.
So people should be careful for what they wish for.

[anon]

There's lots more options if we consider different kinds of reforms. Here are some different options for reform that I prefer.

(1) Change our norms surrounding what an acceptable report turnaround time looks like (1 month max?). As the recent Daily Nous posts illustrates, plenty of reports come from a place of incompetence and arrogance - why are we *also* giving people 3-6 months or more to write these things up?

(2) Change our norms surrounding what a normal acceptance rate looks like. I think our current norm of low acceptance is part of what drives the arbitrary rejections, when they're arbitrary. I don't know what number would be better. Maybe 25%?

[Evan]

I don’t know how ArXiv fully functions. My concerns have to do with the content of the comments and reviews provided once philosophy peer-review move to the kind like ArXiv. I worry that irrelevant, fallacious, and/or bad comments might flood the review. How do we ensure that public reviews are done in good faith? Do other people get to see other people's critiques/reviews?

[aphilosopher]

The Heesen and Bright paper seems to be advocating for something in between abolition and the status quo. If I understand the idea, it's something I've advocated for to friends: people submit to an archive, users comment/rate articles on the archive, and some standard is used to "certify" archived papers that reach some level of positive review. This certification could be human editorial curation, completely algorithmic (e.g., based on the equivalent of "likes" and "down votes"), or some mix of the two.

I don't buy the concern that this will create more for us to read. First, none of us read everything published anyway, so that won't change. Second, nobody will or should feel obligated to treat an archived paper as a "certified" (published/reviewed) one. Third, the whole process (if done well) should create a marketplace of ideas: through sharing, up votes, or whatever, archived papers will filter to the top and catch more and more people's attention. So, just read stuff from your friends, people you follow, and whatever gets up voted. Pretty much what happens now, anyway.

I also am not sold on the bias part. Set aside the concern that famous people will have a leg up. They already have a leg up (because virtually all referees already can spot their work, even blinded). The real question is whether nonfamous people from unfancy places will lose their opportunity. The idea is that blinded review means they have a chance, and the worry is that if it's not blinded, they have no chance because nobody will bother to read their stuff. I tend to think the scale of the crowd is a counter to this. Sure, many people will ignore your stuff, but if there are enough people in the system, the odds are still good that (if your stuff is actually interesting or worthwhile) *somebody* will notice and start up voting you.

I'm also not sure I buy Rowbottom's concerns about the transition. Nobody would advocate that, tomorrow, all journals close shop and we try to immediately open a new slate of journals all based on the new model. Presumably the starting point is just to open one or two journals based on the (or a) new model, and let them compete. Surely the old journals will continue, and we'll see how it all plays out in the long run. The real issue is that we, as a profession, should think about experimenting and setting up these new sorts of journals. The APA, and serious philosophers with clout, should take them seriously and push them as legitimate models of publication.

A genuine marketplace of ideas offers another advantage: it will actually incentivize people to do work that is of interest to a wide range of philosophers. Right now so much philosophy is overly technical, uninteresting, boring, hyperspecialized, and lacking practical value because publication decisions are made entirely by a handful of people with idiosyncratic interests, no incentives for mass appeal, and (often) vicious standards (e.g., think of "reviewer 2"). A crowd-based rating and review system in which papers have to compete for attention will wash out idiosyncrasies of individuals and incentivize writing interesting stuff. (I'm sure there will be enough nit-pickers out there to maintain rigour and professionalization.)

A final point: The journal system as it exists in philosophy today didn't arise out of some objective deliberation 150 years ago over the epistemically best model. It was a solution to the problems imposed by the technology of the time. When physical print media, snail mail, and human labor are your tools for disseminating research and facilitating academic discussion, I assume a system in which people submit to editors who find reviewers to help select papers worth distributing is about the best way to go. But that's not our world now. We have other means of disseminating, reviewing, and curating research, and we should explore them.

[Anonymous]

We could always adopt the law school model and have PhD students run the journals....

[J]

I think we should stick with the current system. If we abandon peer-review this might hurt junior scholars. Established people will get their work read whether they are peer-reviewed or not. But who will spend time reading and engaging work of unknown junior scholars unless they show that their work is good (by successfully getting through the strict peer-review process)?

[Assistant Professor]

Of course to J's point, when senior people with narrow concepts of what "good" philosophy is or ought to be are the ones doing the peer review, this might also disadvantage junior scholars trying to push at traditional ideas, methods, or topics. Not sure about how best to get past gatekeeping. I am also concerned that the people with a lot of twitter followers get the most "view" or alt metrics on their papers and this is also a false sense of value/excellence of work but becomes what gets talked about/noticed.