On Mumford on Peer-Review, and How the Physicists Do It Better

This article on peer-review by Stephen Mumford has been making the rounds, and the following excerpts in particular strike me as worth discussing:

A while ago I was asked by a journal of high standing to referee a paper on a topic of which I had little knowledge. I immediately pointed this out, but the editorial assistant soon afterwards replied that it would be fine as they valued my opinion nevertheless. I was uncomfortable. The journal in question had, during the course of my career, rejected every one of my submissions, now numbering almost a dozen. Their position seemed to be that while they had consistently deemed me unworthy of publication, I was nevertheless worthy of passing judgement on what was good enough for them to publish: outside my own topic too. If I really knew what they wanted, I would no doubt have produced it myself by now. I turned down the assignment.

--------

There is a big intellectual flaw in the peer-review system. It is inherently conservative. Suppose an editor succeeds in securing an ideal referee, eminent in the field and working on the same subject area. Such a reviewer could well have a vested interest in protecting a particular view or theory. Paradigm-changing work is unlikely to go down well with those who support an existing paradigm. Many papers in my own field work within an existing set of shared assumptions, offering only small additions or footnotes, which produce a dull read.

--------

Let us see with each published paper a date of submission, date of acceptance and names of the referees. Let authors know in advance the average decision time. Let us know if asked to revise and resubmit whether it will go back to the same referees or to different ones.

Personally, I think that Mumford's observations on the problems with peer-review are spot-on. However, I don't think his positive proposals would do much at all to improve the situation. How would posting dates, of submission, acceptance, names of referees, etc., correct for any of the problems Mumford cites in his article? Although I hesistate to get on my hobby-horse yet again, I can't help but think that there is already another system out there that does correct for most, if not all, of our current system's problems.  Here, again, is how the physicists do it.  They:

1. Post their working papers on a database called the Arxiv.
2. Read each other's papers, give feedback, and talk about those papers widely, and publicly, on physics messageboards.
3. The feedback they give each other on the Arxiv enables them to help each other improve their papers (which is, like, totally awesome) and, in some cases, disprove and weed out bad papers.
4. The papers that start getting talked about a lot get a "stamp of approval" by the profession before they head out for official peer review.
5. The official peer review process itself is something of a formality.  The papers that have generated a lot of discussion on prominent messageboards (and have not been disproved) are given the green-light by reviewers, and those that haven't are rejected or read more carefully.

Now, this alternative system isn't without potential problems. It is, strictly speaking, possible for bad work to get a lot of attention on messageboards -- though, for what it is worth, when this happens in physics it usually comes out on the messageboards why the work is bad, and the bad stuff doesn't make it into print. Still, it seems to me that, all things considered, the physicists' approach is far superior.  It:

• Takes peer-review mostly out of the hands of unaccountable reviewers.
• Places it instead in the publicly accountable hands of the profession-at-large (as people explain, publicly, on the net, why such-and-such paper is SUPER GOOD or SUPER NOT-GOOD), while also
• Providing forums and incentives for people to help one another improve their papers (which helps everyone).

All of this seems totally awesome to me.  Why don't we philosophers do it? 

Comments

[Lewis Powell]

You didn't note the trade-off that the reviewing is not even remotely blinded on such a model.

[Marcus Arvan]

Lewis: you're right, I didn't. But isn't the trade-off worth discussing? "Blind" review these days isn't all that blind, to begin with -- and might the aims of blind-review be achieved better in other ways, in light of new technology (i.e. the web)? As I've mentioned before (and imply in my post), the practice seems to work splendidly in physics. It's open and transparent in a way that traditional blind-review isn't, it's collaborative, helping authors improve their work, and it seems to vet good *and* bad work effectively. When people read something they find interesting -- whether it is by a well-known name or someone junior -- they tend to talk about it. As I see it, that's a good thing...far better than traditional blind-review, in which the gatekeepers are reviewers who may or may not even read the paper carefully...

[David Morrow]

The line from Mumford that really struck a chord with me is: "Nothing is as infuriating as working on a paper for a long time - sometimes years - only to be told by a reviewer who has looked at it for an hour that it's not been thought through."

I've probably mentioned this before, but I think PhilPapers provides much of the relevant infrastructure. People can post unpublished papers there. Other people can find them and comment on them. The biggest obstacle at this point to implementing the Physics/Arvan method of peer review is professional inertia.

[Marcus Arvan]

David: obviously, that line struck a chord with me as well. ;) Anyway, I think you're right. PhilPapers does provide much of the relevant infrastructure. It would be nice if someone over there made a push in favor of developing a strong and easy-to-navigate messageboard community where newly posted papers might be read and discussed. I know they have messageboards already -- but perhaps they could develop and emphasize that aspect of the site quite a bit more. They have this great resource: a running list of newly posted papers. How cool would it be if they really made a push to develop the communal/messageboard part of things?...a place where people across the profession regularly visited and discussed new papers en masse. I, at any rate, think it would be very cool, for a variety of reasons.

[Brad Cokelet]

I don't think the peer review system is flawless, but I really worry your proposed alternative will favor those who come from good schools and have good connections. When I review I resist the temptation to look up the author and I think this is a very good thing. I worry that on your system people with connections would get lots of heat in the discussion zone just because of who they know and that this would lead to their papers getting an easy pass. It seems like people would weigh discussion by "big name" people and such more too, and this would only make the problem worse. It is possible I am idealizing how little of an effect this already has, but I am glad there is at least an effort to give Jonny Bad Personality, from No Name Tech a shot at publishing in Ethics, JP, or Nous.

[Marcus Arvan]

Brad: I certainly get your worry. It is a serious one, to be sure. But I'm not sure about it myself. In light of what I've seen and heard in the physics case, I guess I'm more pessimistic about the merits of traditional peer-review, and more sanguine about the likelihood of good work being recognized and discussed as such on public boards, than you are. In physics -- including not just experimental physics, but abstract theory which is much closer to philosophy -- "nobodies" with great ideas, including independent researchers, have often rocketed up the charts (in terms of paper downloads) and online discussion. In other words, in physics at least, I think opposite of what you suggest has actually occurred. "Nobodies" have a *better* change of getting noticed and discussed, not less. And I'm not sure there's reason to expect any differently in philosophy. I guess I think -- both in philosophy and physics -- people tend to flock towards ideas they find interesting and potentially fruitful.

[Brad Cokelet]

Well that does sound appealing as an outcome!

[Richard Brown]

I think this was the idea behind Andy Cullison's Sympoze (http://www.sympoze.com) but I don't know what ever happened to it...

[Marcus Arvan]

Richard: Sympoze, as I understand it, is (was?) quite different than the physicists' model -- and, I think, much harder to get off the ground. Sympoze's aim was to replace traditional peer review with crowdsourced reviews. The physicists' model is different, and works within the traditional institutional framework. Physicists still have to send their papers to journals for review. It's just that the papers are already so widely discussed on messageboards that reviewers know whether to accept the manuscript the minute they receive it (as they've most likely already read the paper on Arxiv and seen whether others in the profession accept the paper as important. I think this is a much better approach for reasons of institutional inertia. Whereas Sympoze aims to start a whole new enterprise (a profoundly difficult endeavor to pull off), the physicists' model just supplements the traditional peer review model with widespread online discussion -- something I expect the people over at PhilPapers could probably pull off easily if they made a real push towards revamping their messageboard system and forefronting it on the site.