Steps for changing peer-review?

I've been arguing for changes to our system of peer-review for about six years now--specifically, in favor of the more "open" peer-review utilized in math and physics, where papers are posted to a public repository, discussed openly, and routinely cited even before being sent to journals. I've tried to catalog the benefits I think such a system would have in philosophy--for authors, reviewers, editors, and the profession. Further, some researchers have defended a similar proposal in more detail, in part by using empirical research on peer-review. To clarify, my argument is not that our current system is terrible in absolute terms or "broken" (though at least one editor has raised serious concerns about the viability of the status quo). Rather, it is merely my belief that we can probably do better--and that the math and physics is probably the way to do that.

I recognize the case that I and others have made for changing peer-review is still controversial. However, while not everyone agrees, one question that arises for those who it should is: what steps might individuals and the profession more broadly take to realize change? Peter asked this question (and more) in a recent comment, where he wrote:

[L]et's suppose that you are correct. How do you think we might begin transitioning as a discipline to something like what you suggest? Do you think it might start best with some subdisciplines beginning something like this (I know philosophy of science already has something a bit like this)? Do you think this would work better than trying to get something together discipline-wide (perhaps through PhilArchive, for example)? It is an interesting problem since it involves both a change in behaviors and norms on the one hand and the creation of an infrastructure that is designed for this purpose.

I've been thinking about these issues a bit, as change can only occur to the extent that people begin to take action. What types of actions might we take? What types, if any, should we take? Allow me to explore a few possibilities and then open things up for discussion (i.e. both further brainstorming as well as critical discussion).

One obvious initial question is what you, I, or any other member of the profession can do to help foster change. Here are a few things individuals might do:

1. Begin openly posting preprints online.
2. Try to foster discussion of preprints (perhaps using philpeople's"sessions" function to solicit feedback on papers).
3. Try to disseminate such discussion, drawing awareness to it.
4. Make it a practice of citing preprints relevant to one's work (as is standard practice in math & physics).

There are, I think, ways the Cocoon and other blogs could help. One idea I've been toying with is a weekly Cocoon post listing new unpublished manuscripts uploaded to the site, followed by some discussion (i.e. me sharing thoughts on which papers I found interesting and why, as well as open discussion in the comments section). While this might only be a drop in the bucket (as it were), it could be one way this blog could promote change. What does everyone think of that idea?

While I think these may be some good ways to start, I also think a real change in norms is probably more likely to occur the more that individuals and institutions of professional influence endorse and promote relevant changes. I expect that if this occurs at all, it is likely to change slowly. But what changes might help?

One (perhaps remote) possibility here would be for the individuals who run philpapers/philpeople/etc. (David Bourget & David Chalmers) to endorse or incentivize the new norms. Given that philpapers seems to be pretty much the repository for philosophy these days (it is the place I visit the most often to find new papers), it seems like a natural place to play the role that the ArXiv does in physics. What could philpapers do? Aside from explicitly endorsing the ArXiv model, they might introduce new functionality. For example, papers on the physics ArXiv contain a link to all blog discussions of preprints (see the right-hand side of this page). This kind of functionality would help readers discover which new preprints are being discussed, and how they are being received prior to peer-review. It would also, I think, incentivize blogs to discuss new papers! Similarly, while philpeople has (or so I've seen) a "session" function where people can invite others to read and give feedback on their papers, from what I can tell the current model is akin to academia.edu's - one needs to invite people to the session (and the session is otherwise "closed"). Another possibility here is that people might be able to create open sessions where anyone could read and provide feedback on papers.

One final (perhaps more remote) possibility is for journal editors to avow the "open" peer-review model--perhaps in an editorial statement and/or their journal policy (which might encourage--or at least not discourage--authors from openly posting and discussing unpublished preprints). While it may be unlikely at this point for (m)any editors to do this, at least one editor has already argued our current system is "emergency status"--and I've heard at least one other editor say our current system is "broken." If there are indeed editors out there who think that the current system is unsustainable--because there are too many unready or unpublishable papers submitted each year (swamping the journals, outstripping their ability to recruit referees, etc.)--then I think it may be worth them considering endorsing and incentivizing new practices. For, as I have argued, an "open" peer-review system like that of math and physics may indeed plausibly lessen the load that journals have to bear--as open discussion of preprints can give authors a better idea of whether a given paper is publishable or hopeless, in need of more revisions before formal submission, and so on. I don't know about you all, but one of the reasons I submit some of the papers I do (even ones I later find out "weren't ready") is because, in our current model, everyone is incentivized to rush things out as quickly as possible (given how long turnaround times are, authorial worries about "getting to market first", and so on). 

Anyway, these are just a few thoughts. To be frank, I'm just sort of brainstorming at this point. But I wonder what readers think. Would it be good for the Cocoon to have a weekly posted devoted to discussing new preprints? Are the other things I've suggest realistic or desirable? Are there other, better ways to promote change? I'm curious to hear your thoughts!

Comments

[Amanda]

I know a lot of people are afraid to post papers online because of ruining blind review. Of course, I have argued it is commonly ruined already. However there might be a really awkward phase of starting this when peer review is the official system, as it will make it more likely that blind review will not be blind.

So how is the system you imagine Marcus? After a wide range of people discuss a paper unpublished, then this person submits a paper to a journal and it is reviewed, non-blindly, by one or two people? What is the point of publishing if everything is already online and discussed?

[Trevor Hedberg]

The bulk of the problem with the current peer review system boils down to journals being overwhelmed with too many submissions. (That is actually the only issue that Velleman discusses in the Daily Nous post of his that you link to.) That's what leads to long turnaround times and most other undesirable outcomes. Any full discussion of reform should consider whether there would be ways to alleviate this problem without transitioning to a full-scale overhaul of the system that's currently in place. Velleman's own suggestions involved essentially not allowing graduate students to publish at all. I echo the thoughts of many commentators on that post in thinking that this solution would be a step in the wrong direction, but it's at worth considering what options for reform are available.

Neil Sinhababu's proposal was to significantly increase the number of available journals. Details here: http://dailynous.com/2016/06/07/2000-spaces-for-10000-papers-why-everything-gets-rejected-referees-are-exhausted-guest-post-by-neil-sinhababu/

I imagine there are also other directions we could go, though I cannot recall any other open discussions about the possibilities right offhand.

[Marcus Arvan]

Hi Trevor: thanks for weighing in!

I share the concerns others have raised about Velleman’s proposal. But I’m not convinced that Sinhababu’s proposal is a promising alternative. There are already a lot of journals. The problem, it seems to me, is that regardless of how many journals there are, people will tend to swamp the same top journals with the vast majority of submissions.

I think there are two reasons for this. First and most obviously, people want to get their papers into the top journals—to get a job, tenure and promotion, reputation, and so on. But I don’t think this is the sole cause. A person is probably going to submit a paper to a top journal only if they think there is some chance it will get in. I don’t know about you, but if I was very confident a paper would never get into a top journal, I would shoot lower (indeed this is what I sometimes do).. But here is what I think happens in our current system. People don’t get a ton of feedback on their work before sending it out. They think to themselves, “Maybe my paper will get into Mind or PPR, so I’ll give it a shot.” The problem then is, when hundreds of other philosophers think this way, they *all* end up sending their papers to the same top journals—swamping them. So again, I don’t think it is likely to matter how many journals there are. The existing structure and incentives of our current system predictably lead people to all send their papers to the same few journals.

I think a transition to the “open” math and physics system would help solve this swamping problem in multiple ways. First, authors would be much more likely to learn from online feedback to their public preprints whether they are publishable at *all*—and end up submitting fewer papers altogether. For my part, I have learned through anonymized review that some of my papers were unpublishable. In an “open” system, I might have learned that without ever submitting my paper to any journals at all (as I might learn from online discussion that my paper has fatal problems, etc.). Second, online discussion of preprints would plausibly give authors a better idea of whether a given paper is viable for a top journal, or whether they should focus on lower-ranked journals. For example, one might learn that one’s paper isn’t as original as one thought, or has problems that are likely to lead to rejection at the very best journals—but one still might be willing to give lower ranked journals a shot rather than give up on the paper. Third, another reason journals are swamped is because of R&R verdicts—papers that sometimes need to go through two or even three stages of revisions before being accepted. An open peer review system would plausibly reduce the number and difficulty of R&R’s by virtue of authors getting better feedback on preprints (and revisions of preprints) online before journal submission.

Of course, this is speculative—but I think when compared to the alternatives (Velleman’s and Sinhababu’s proposals), there is more to be said for this one. Part of the reason I think this is that I have witnessed my own behavior as an author. There are some papers I wish I had never bothered submitting, only because I learned through peer review that they were unpublishable. In an “open” system, I think I probably would have never bothered submitting some of those papers.

The other part of the reason I think this is my experience as a reviewer—reviewing papers that seem to me obviously unpublishable, and which the author might have learned were unpublishable (or at least might have been vastly improved) if they had received more feedback from more people prior to peer review...which they plausibly would have learned from online feedback.

[Trevor Hedberg]

Marcus -- you may be right, but I think determining what solution is most practical is complicated. There would be significant transition costs to such a dramatic overhaul (as there always are when making a radical reform). I am particularly uncertain what the impact would be for early career scholars. Would peer-reviewed publications become somewhat less valuable? Would people who had stronger personal connections with big-name philosophers be more likely to get their pre-published work notably discussed, especially before this system became a universally practiced norm? Would this means of getting feedback on one's work disincentivize people from submitting to conferences as a means of getting similar feedback? If so, to what extent would that be a bad thing?

Thus, it's worth considering whether the current system could simply be tweaked or modified to ameliorate the problems. The Velleman and Sinhababu proposals are examples of that since they leave the basic structure of peer review largely in tact. I suspect that a milder change to the status quo would be easier to implement and also be more likely to gain widespread support from the profession at large than a complete overhaul of the whole system.

[Peter]

I know I am late to this conversation, but I wanted to chime in.

I also apologize for the length of this comment…it just kept growing.

There are some interesting worries raised in some of the comments, but I want to leave off discussing them at the moment to focus on the heart of Marcus’s original post. (This might seem odd since I am setting aside the question of whether we should do something to discuss how we might do it. I think further discussion should take place before anyone takes radical steps to transition things, but that is a matter for other posts.)

Marcus brings up some interesting things individuals might do to nudge the profession toward the model he proposes. So, for example we might start posting our own papers online before submitting them to journals; we might openly discuss the preprints others post; we might draw attention to the fact that we are discussing them; we might begin citing the preprints of others.

I think that each of these could nudge the norms of our profession in the direction that Marcus proposes. I can think of three worries with doing some or all of these:

(1) There is a worry that if we post our own work, we will hurt our chances of having our papers accepted at journals, since editors might not be happy either (a) that the work is already readily available or (b) that the blind review process might be impossible. Alternatively, we might just be worried that if referees knew who wrote our papers, they would take our work less seriously if we are not big shots. Additionally, by talking about the early work of others, we might bring these harms upon them.

(2) That whatever problems our system has, we would make things worse by nudging professional norms in this direction.

(3) That discussing the pre-acceptance work of others brings with it opportunity costs. Any time we spend blogging about the work of others is time we don’t spend writing, teaching, or just hanging out with our families.

(2) involves the issues I set aside here. I am assuming that (1) and (3) are not problems with Marcus’s model, but rather with taking certain steps to help the profession transition to that model. (3) turns on the value we might find in blogging about the work of others and on other aspects of our lives that vary widely (like how desperately we need to fill out our CVs or how little time we get away from work). Importantly, the value of blogging about the early work of others is likely to increase if the new model catches on, so the worry concerning it is greater before the model is widely accepted. (1), I take it, is a worry for many. Many people seem to worry about referees discovering identities by googling parts of their papers, and posting them online would certainly cause problems here. One might upload them with nonsearchable text (as scanned images or something), which would make it harder for referees to find your work, but won’t stop them from knowing your identity if they already read your work.

Whatever the value of these worries, I think they will stop a large number of people from doing as you suggest. I do wonder whether it would be easier to change things if one tried to start changing the norms of a relatively small sub-discipline, especially one with some distance from math and physics. I think that if some of these changes were to catch on even somewhat, and the effects of this change were not too bad, it would be easier to get others to change their practices as well.

Some of our subdisciplines are, perhaps, small enough that even a few people, even early career scholars (as long as they were productive enough) might get this ball rolling. If a small group of, let’s say, people in aesthetics or philosophy of religion or something started to post all their work publically, invite others to do so, link to the online work of others, and actively blog about such early work, I could see others following suit. Eventually this could provide an attractive and relatable model for other philosophers to follow suit.

Of course, one might say that there is no reason we cannot just model ourselves off of physics or math, but there are a couple barriers to this. First, many might worry that there are significant enough differences between these fields, if not methodological, then perhaps sociological, that might give us reasons to think that this model would not work for us at this point in time. Even if this worry doesn’t arise, seeing this model work in a philosophical subdiscipline, especially one that is not closely tied to mathematics or physics, would make it easier for other philosophers to see themselves changing their own practices.

[Nichole Smith]

Marcus, in response to the problem you identify with Sinhabahu's proposal, might the solution be a move away from general journals? My understanding is that the top journals are mostly generalist, and among specialist journals, the top of them are general within the specialty. Of course, generalist journals have a greater number of papers that could be sent to them, so the best of them get the most.

But what if there was no truly general option? Perhaps a "Miscellaneous" journal or dozen could exist for those papers that really just do not fit into a specialty, but the vast majority could go into specialty journals. And if the top specialty journals are still seeing too many submissions, they could further specialize.

I imagine this falls somewhere between just increasing acceptance rates and a switch to the arXiv model in terms of complexity in overcoming inertia. If there's a need that fully general journals serve that a constellation of specialty journals couldn't, I'd be interested to hear it.

[Marcus Arvan]

Trevor: you're probably right that the other alternatives are more likely to catch on--but as I've noted, I'm skeptical the more mild alternatives are likely to do much to solve various problems with our current model. In terms of your other questions (viz. effects on early-career folk, the value of publications, etc.), my sense is that in math and physics publications still matter (viz. jobs, tenure, and promotion), but that more important still is the level of uptake with one's work. For reasons I've explained (viz. quicker dissemination of one's work, greater online discussion, etc.), I'm inclined to think the math & physics model is likely to lead good work to be recognized more quickly--something that I think would benefit everyone, including early-career people (who currently have to wait months or years to get their work out there due to 'anonymized' review).

Hi Peter: Thanks for raising these issues. I worry about (1) myself. I'm not as certain about (2) or (3). You're certainly right that there are opportunity costs involved--but of course whether those costs stand against the proposal all things considered is a matter of weighing them against the benefits of the new system (which I've suggested are likely to be considerable, though of course I could be wrong!).

[Marcus Arvan]

Hi Nichole: I think we've probably seen the number of specialist journals explode in recent years for more or less these reasons. However, I'm not sure they really solve various problems with our current model (which I've argued are endemic to anonymized review in general). I'm also a bit wary of overspecialization in philosophy in general and how increasing the number of specialist journals further might add to that. But this is admittedly a parochial concern of my own.