Should COVID spur peer-review reform?

Long-time readers of the Cocoon may recall that I have advocated for reform to our discipline's peer-review system for about eight years now. In brief, our current system of peer-review seems to serve no one well. As I argued here and an informal survey I carried out seemed to support, our current system of anonymized peer-review:

1. Overwhelms editors with too many submissions.
2. Irritates and overwhelms reviewers with too many 'half-baked' submissions.
3. Frustrates authors with long and inconsistent turnaround times, peer-review reports of dubious quality, etc.

In addition to each of these problems, our current publishing system almost certainly impedes the timely dissemination of new research. COVID-19 is a prime example of the problem here. To see why, consider the following offhand remark that Jacob Stegenga makes in a guest post at Daily Nous today:

So far in the lockdown—May whatever-it-is-today—I am not aware of a systematic piece written by a philosopher of science on the pandemic or the policy response (if you know of one, I would be grateful if anyone reading this would direct me to it). Of course, I do not mean a journal article—our journals move too slowly for this—but I mean something deeper and more impactful than what normally appears in social media or blogs.

This is particularly striking to me. Over the past two months as the COVID-19 pandemic has progressed, articles in medical journals have been rapidly published (see e.g. here and here). Yes, there have been some COVID-related articles published in a few niche ethics journals (primarily interdisciplinary ones). But, as Stegenga's post illustrates, philosophy has by and large lagged behind other fields in disseminating new research on this world-changing topic. How many philosophy articles on COVID are stuck in anonymized review right now, as this pandemic rages around us? One can only imagine. Are there dangers to the rapid dissemination of research? Sure, particularly when it comes (as we all know by now) to epidemiological modeling. That being said, our current system of anonymized review--in addition to all of its other problems--is systematically poorly designed for this moment and world-changing moments like it. Philosophy should be on the front end of COVID, examining the myriad questions it raises in philosophy of science, ethics, etc. And yet, so far at least (as Stegenga notes), we seem to be decidedly lagging behind other fields.

Is it possible to go outside of our current publishing system to disseminate research more quickly--for example, by posting papers to the philarchive or publishing public philosophy in The Atlantic, etc.? Of course it's possible. But why should philosophers have to choose between these options when other fields don't? In physics, everyone expects unpublished drafts to be posted to the ArXiv prior to journal review. The same is true in medicine with the MedRxiv. In philosophy, on the other hand, there are no clear norms here: it's unclear from author guidelines at journals whether editors or reviewers frown on authors posting unpublished drafts to the philarchive. Indeed, for my part, I mostly avoid posting unpublished drafts there for this very reason. Will a journal editor or reviewer think it violates anonymized review? I have no idea, so for the most part I just avoid it. But why should this be how things are in our profession, when other professions have been using a more public alternative for decades?

The way I see, the time for peer-review reform in philosophy is now. How could such reform be achieved? Here's one possibility: the APA could take a clear stance in favor of authors posting unpublished drafts on the philarchive, recommending that journal editors and reviewers not hold it against others. And here's another possibility: journal editors could come out and state clearly that their journal endorses the practice. These changes, I believe, would benefit us all. A clear disciplinary norm that says it's just fine for authors to post unpublished drafts on the philarchive would incentivize authors to do so, enabling them to get more feedback on their papers, plausibly lessening the number of 'half baked' papers submitted to journals, reducing the number of revise-and-resubmits authors have to go through, and disseminating new philosophical arguments more quickly. We don't live in the 18th century anymore, and shouldn't keep using a peer-review system designed for that era. We live in a vastly different era, one where philosophers could make more timely contributions to public debate...if only we finally got down to business on peer-review reform.

Or so say I. What say you?

Comments

[Grumpy]

I'll bite first ...
I do not think we should encourage archiving, as in other disciplines. I publish in two other disciplines ... and a co-author in one has archived pre-prints before. So that is me.
I think philosophy papers in draft form are often - far too often - too drafty. I predict they will later be used against candidates, when they are on the market. How do I know they are too drafty. I read too many of them as a referee. Why is this not a problem in other disciplines? In part because there are more objective criteria for evaluating contributions. One knows if their paper is making NO contribution whatsoever, or whether it is near-incomprehensible.
People need to cultivate professional relationships with people who can give them feedback. The pandemic and lockdown is reminding me how important real human contact is. I am a shy person too, Marcus. But if you want a career in philosophy you need to get feedback from people who know the field, and you need to learn to respond to it. And you need to have enough restraint to hold onto your papers until they are quite polished.
Finally, I really do not believe that people in the profession are sitting around reading archived papers by people they do not know or have never heard of. Each reader of this blog should ask themselves: how many papers have they read in the last 12 months? and then how many UNpublished archived papers by people they have never heard of before, have they read in the last 12 months? I bet the ratio is close to 100:1.

[WorriedJuniorScholar]

I'm skeptical of the verdict that the current system serves no one well. Anonymous review is one of the only (and most effective) protections that junior and less-famous scholars have in order to guarantee that our work will be taken seriously on its merits. The current system of posting drafts to personal websites removes some of that protection. Posting drafts to philarchive removes all of that protection.

I understand that the current system of peer review has many problems. I don't know if I would like to claim that the importance of maintaining anonymity trumps these problems. But there are identifiable constituencies who certainly benefit from the current system.

[a philosopher]

I think we have to disentangle the archive issue from the speed issue. I regularly see published work in the sciences that goes from initial submission to revision in 1 month, then from revision to publication in another month. (This data is normally in some date log printed on the first page.) Even the fastest journals in philosophy don't move that fast. I see a journal that can reliably go from submission to publication in 6 months, in philosophy, as super fast.

Why is it that science journals can handle this turnaround, but most philosophy journals operate more so on a scale of 1-2 years? If we had this type of turnaround, we could already be seeing the first published work in philosophy on COVID-19, and we'd see a bunch of articles coming out over the summer (assuming people were writing them), in time for the fall semester.

I know Marcus is worried about even faster times, e.g. on the scale of a week or two, so that we could react in near real-time and enter the public discussion. Getting our embarrassing journal turnaround times down to "industry standards" wouldn't help. But I suspect we could achieve almost what Marcus wants, even without archiving, with the right mindset and motivation.

Imagine, for example, if philosophers were looking to write these COVID19-relevant articles, and if editors and referees were looking to fast track them. Journals could have put out special calls for papers, promising turnarounds of 1-2 weeks, or simply kept an eye out and prioritized COVID19 submissions. I'm guessing this whole idea sounds laughable to some people, and that's precisely the problem. The culture of peer review in philosophy breeds this slowness. People expect to be given a week just to respond to referee requests, or given 3 months to complete a review. (I know lots of people are *much* better about refereeing, but lots are really bad.)

I just don't see why, had we as a discipline wanted to, we couldn't have prioritized our response to COVID19-related philosophical issues and work, getting journal articles out in time to be relevant. Judging by all the social media philosophizing I see happening, we philosophers certainly have a lot to say.

[Grumpy]

A philosopher,
This is Grumpy, again. I agree. In the post above, I mentioned I publish in two other fields. I just wrote paper, which I started to write in mid-March 2020. It involved collecting new data, and analyses, etc. It has been reviewed, resubmitted twice, and accepted for publication. One of the resubmissions required additional data collection and analyses. I expect to get proofs soon. That is NOT in a philosophy journal. It can be done. It is largely a (academic) cultural thing.

[a philosopher]

Hi Grumpy, thanks for that example! That's exactly what I mean. I don't see why we can't achieve those results in philosophy, aside from unhealthy social norms around reviewing and our unwillingness to change them, or even conceptualize something different. (And whatever excuse you can come up with for why it takes you a week to respond to a referee request, or 5 months to review a paper, don't you think those in faster fields can say the same thing? --- and they still get it done!)

Fwiw, I think this sociocultural issue goes beyond review, just to our day-to-day work habits (many of which I'm guilty of as well). My friends in the business world, for example, tell me that it's just expected you will reply to emails the same day, or even within a few hours (during 9-5), even if the email is noncritical --- it's just polite. Many of us philosophers (myself included) are guilty of sitting on emails from colleagues for days, or weeks even. Yeah, yeah; I know we're all busy. But so are my friends in the business world. If they can get it done, so can we. It's a cultural thing.

[Amanda]

Whenever we talk about peer review publications, everyone agrees there are lots of problems, but then people comment that we should keep peer review because it is egalitarian, i.e., it is the only thing that allows unrecognized scholars from not prestigious places to make a name for themselves. I am skeptical of this because:

1. Well, let me say NOT because I think it's false that people from not prestigious institutions often use peer review to their advantage and publish in great places. This does indeed happen. And this IS a good aspect of peer review.

2.However, I don't think the advantages are nearly enough to overcome the disadvantages. Because, for instance, we all know of "not prestigious" people who have published is great places, and yet have been on the job market for years on end in temporary, insecure, positions. So it is not at all clear that non prestigious people publishing in great places really provides that much of an edge. I think it gives pride to the publisher, but often, not much else.

3. Those with connections still have huge advantages, because as myself and others point out often, blind review is often not blind. The more prestigious one is, the less likely it is to be blind. So even if blind review allows not prestigious people to publish in great places, I think this outweighed by the advantages given to the connected. It allows the connected to say, "I have all these publications by merit," when often, they do not.

4. Perhaps I am too optimistic, but I actually think an archive system would be better for the not connected. I do not think work would be ignored because someone was not famous. I think it might be more like facebook, in which I see very "popular" facebook philosophers who often ask interesting philosophical questions or write interesting philosophical posts, and gain a reputation this way. Persons always comment and reply to their posts, etc. I think that not prestigious people could gain a reputation much faster with an archive system than with a peer review system. I could be wrong, of course, but with ALL the problems with peer review, I think giving it a test run makes sense!

[a philosopher]

In line with Amanda's skepticism, I'll also say that the lack of blinded peer review doesn't seem to stop lower prestige people in the sciences from making a name for themselves. (At least, the existence of an archive system doesn't stop them.) I mean, I'm not intimately acquainted with mobility in other disciplines, but I don't think (e.g.) physics or psychology are any *worse* w.r.t. opportunities for junior and low-prestige researchers.

Let's call this the "parochial fallacy": reasoning that we can't do X because it will lead to bad consequence Y, in the face of lots of evidence that people outside your circles do X without Y.

A final thought: even in principle, blind review seems like at best a poor bandaid for the real problem. Early and low-prestige philosophers face a challenge because, largely as a culture, we philosophers have cultivated a bias against them. We fetishize names like "Princeton" and "NYU" and indulge in the idea of "geniuses" far too much. People are far, far too willing to take a chance on an ABD student from a top-ten program who has one sort of interesting idea in one paper than they are to hire a 3-yr-out PhD from the bottom half of the PGR with 8 solid papers published. If people would cut this crap out, we wouldn't need blind review.

[WorriedJuniorScholar]

Amanda and a philosopher are quite worried about prestige bias in hiring. I am too.

Amanda is also worried that blind review isn't as blind as we'd like it to be. I am too.

I'm not sure that the evidence from the sciences tells the same story that a philosopher would like it to tell. Junior scholars in the sciences publish all of their papers as graduate students and postdocs with senior PIs as coauthors. And likewise, most papers by these senior PIs are coauthored with grad students and postdocs. So there's not the same opportunity to discriminate against papers authored only by junior scholars or in favor of papers authored only by senior scholars.

The dack is crushingly stacked against junior scholars in the sciences in at least one area: application for grant funding. The average age at which scientists receive their first major grant continues to rise to the point that many junior scholars have difficulty funding their labs. This isn't necessarily a kind of discrimination (it can make good sense for grant funding agencies to fund established labs) but I don't think we should take it as a sign that having their identities and credentials known makes things easy for junior scientists.

[Sorry, but ...]

Let's call this a fallacy ...
"I mean, I'm not intimately acquainted with mobility in other disciplines, but I don't think ..."
I do know a bit about mobility in other disciplines. One study suggests that where one is at year 10 post-PhD in the sciences and social sciences is where you will be the rest of your career.

[a philosopher]

Since a similar 10-yr study would presumably show similar results for philosophy, I don't think it hurts my point.

The point about grants, from WorriedJuniorScholar, is a good one. I also like the coauthoring point, which seems good to me as well. I'm, of course, ready to concede that the picture is much more complicated than I suggested.

Still, I'm worried that these two responses have already turned my point into a strawman. I did *not* say that "having their identities and credentials known makes things easy for junior scientists". What I said was that, as far as I know, early career people in the sciences did not face more negative bias than they do in philosophy.

If these disciplines can employ preprint archives without resulting widespread discrimination in the publishing/hiring process, then such fears in philosophy are probably misplaced.

Maybe the coauthoring point really does completely undercut my reasoning. Maybe there is something else substantially different that makes a difference (e.g., someone above mentioned how there are less clear standards for what makes a good philosophy paper). Still, I think the positive experiences of other disciplines with preprint archives suggest assuming, at the start, that they will be a net positive in philosophy.

I think human psychology is messy enough that we should be guided by nearby examples of practices more than by how we guess people will respond.