Restrictions on replies: a barrier to philosophical discussion?

A while back, I noted some disturbing facts about citation rates in the humanities. For instance, according to this study:

• More than 90% of published humanities papers have received no citations two years after publication.
• About 82% have never been cited five years after publication.
• Citation rates in the humanities have been consistently this low across the past several decades.

Kieran Healy's recent research on citation rates in philosophy broadly indicates similar trends. As he put it, "on the average hardly anyone is getting cited, be they man or woman." These trends are in marked contrast to the sciences, where (according to the first study linked to above):

• Over 50% of natural science and social science papers receive at least one citation within two years.
• About 70% receive at least one citation after five years.
• Citation-rates in both areas have climbed dramatically and steadily over the past several decades.

Why are citation rates so low in the humanities, and in philosophy specifically? And is this a problem? If so, what should be done about it? Obviously, these are very big questions, and we cannot hope adequately address them here. Instead what I would like to do is focus on a particular issue I have some concerns about: restrictions on "replies" to articles in other journals.

I have sometimes heard people float the hypothesis that citation and discussion rates are low in philosophy because comparatively few papers actually warrant discussion. This is something I have heard surprisingly often (i.e. I have heard more than a few people say things like, "Most published papers in philosophy are bad - so it is no wonder they are not discussed"). Although I deeply disagree with this suggestion--from my vantage-point, there appear to be too many good papers that are underdiscussed, not too many bad papers that are discussed--I am not sure we can evaluate the hypothesis without considering possible confounds: that is, other possible explanations of why so few papers are discussed. I want to suggest that the common editorial practice that many journals have of not accepting "replies" to articles in other journals may be just such a confound.

For those who may not know what a "reply" piece is (sometimes they are also called "discussion notes"), a reply is a publication that primarily, or exclusively, addresses another paper, aiming (for instance) to refute or otherwise undermine its argument. This is in contrast to what are sometimes called "free-standing" or "stand-alone" papers--papers that focus primarily on developing an original positive argument of their own. 

My first two publications were replies, and occurred in one of the few philosophy journals out there (JESP) that actually accepts replies to papers in other journals. However, most journals do not accept replies to papers in other journals--which basically means that if you write a reply, you have approximately one venue, and one venue only, that you can submit it to for publication. Allow me to explain why I think this plausibly erects several related barriers to philosophical discussion.

1. The False-Negative Problem

The first concern I have about the practice of journals not considering replies to articles in other journals concerns the prospect of "false negatives"--that is, good/important replies never being published. Here is how I think this problem arises. If one can submit a reply to only one journal, whether the reply is ever published all comes down to the judgments of one set of editors and/or reviewers. Yet, as we all know, the peer-review process can be "noisy." What one set of reviewers or editors at one journal judge worthy of rejection, reviewers or editors at another journal may judge to be publishable and important. 

Consider for instance some remarks by Jason Stanley reported a while back at NewAPPS:

I'm reviewing Kieran Healy's citation data, and it reminds me again how weird journal acceptance is. My book *Knowledge and Practical Interests* is the fifth most cited work of philosophy since 2000 in Phil Review, Mind, Nous, and the Journal of Philosophy (book or article). Yet the book itself is the result of three revise and resubmits, and finally a rejection from Phil Review. One of those drafts was also rejected from Mind, and also from Nous. All of those journals have accepted papers discussing, in many cases very centrally, a work those very journals have deemed unpublishable.

I have heard many stories like this, of influential papers that were evidently rejected (and in some cases, desk-rejected) at multiple journals before being published (I seem to recall hearing somewhere this was true of David Chalmers' and Andy Clark's famous article in Analysis on the extended mind). 

Because the peer-review process is so noisy/inconsistent, I think it is plausibly critical to the integrity of the process that authors have multiple chances to submit their work at different venues. If one can only submit a given piece of work to only one venue (as in the case of replies), then it makes the fate of the piece (i.e. whether it is published) come down to the (potentially mistaken, biased, etc.) judgments of one set of individuals, who may well reject a perfectly good/important paper. It is only by allowing authors to submit pieces to multiple journals sequentially that "false negatives" (good papers never being published) can be reliably avoided. 

And indeed, it is generally recognized in many different areas of human life--in politics, corporate book-keeping, etc.--that transparency is important. In order to ensure that a system is working as it should, there must be some transparent way to check how the system is working. When it comes to the normal process of submitting articles for review, the process is relatively transparent in this regard. Although sometimes our papers are desk-rejected (a non-transparent decisionmaking process), over the course of multiple submissions to multiple journals the process becomes increasingly transparent. For example, if your manuscript is desk-rejected at many journals, that is some relatively transparent reason to think the manuscript is unpublishable in its present form--as it shows that multiple editors at different journals all came to the same decision. Conversely, if your manuscript is not desk-rejected at some journals--and particularly, if one receives reviewer comments--then, at least over time, one typically gains some insight as to the quality of one's paper: one may get negative reviewer reports explaining why one's paper isn't publishable, or perhaps some positive referee reports suggesting that it is publishable (or at least may be with some revision).

In contrast, the "no replies to articles in other journals" rule makes the publishing process for replies decidedly non-transparent--as something like the proverbial "black box" that one cannot see into. After all, if your reply is rejected without explanation, and you can only send it to one journal, you basically have to give up on it, and may be left with no idea why the paper was rejected. This is for the simple reason that, in contrast with normal articles--which again, one can send many places--with any given reply one can only get one data-point: the decision of the single journal to which one sends it. But one data-point gives very little information, especially if (as is often the case with replies) one's paper is rejected without comment. 

2. The Disincentivization Problem

In addition to the false-positive problem, the rule against publishing replies to articles in other journals plausibly raises a second barrier to philosophical discussion: it disincentivizes people from writing replies to being with.

As I mentioned above, my first two publications were replies. Not only that: I very much enjoy writing replies. Like many philosophers, I "enjoy me a good refutation"! And indeed, critically evaluating arguments is one of the most central and important skills we learn as philosophers. Philosophical discussion only benefits from critical engagement with people's arguments. It provides a clearer picture of which arguments and theories are good, which are bad, which can be improved, etc. Yet, although I very much enjoy writing replies, I basically stopped writing them. Why?

The answer is simple: given the rule that one can only submit them to basically one journal (the journal the original article appeared in), I found that writing replies mostly isn't worth my time. For a while, I wrote a few replies that I thought were strong. But, when they were desk-rejected without explanation, I couldn't help but think to myself, "Why bother doing this? Why bother writing refutations that I have only one shot of publishing, and which may be rejected without any explanation?" Writing replies takes time. Given that stand alone articles can be submitted to many journals, once I started publishing standalone articles, I simply figured my time would be better spent on them, not replies. Hence, I don't write replies. I have been disincentivized from writing them.

I suspect I am probably not alone here. What incentive is there for anyone to write replies when the costs of writing them are significant, but one only has one shot to publish them? This problem is only compounded, I think, by the problem of non-transparency mentioned above. Given that replies are sometimes (often?) desk-rejected without comment, getting a reply rejected can be a very frustrating process--one that leaves an author with no real idea why their paper wasn't published, and without any real recourse (such as submitting the paper elsewhere) for determining whether the decision arrived at was correct. If anything is a dispiriting disincentive, it seems to me, that is. Why write and submit a reply when (A) chances are, it will be rejected (since most papers are rejected!), (B) one can only submit it one place, and (C) one may get no transparent explanation for whatever decision is reached at the one place you can actually send it?

3. The Misapplication Problem

I also think there may also be some reasons to worry whether the editorial rule against publishing replies to articles in other journals may be misapplied, in a manner that plausibly erects a third barrier to philosophical engagement.

Generally speaking, "replies" are very short (2-3K words) and purely negative in nature, critiquing a particular paper. However, I have some experience with papers that I do not think are properly considered replies being desk-rejected. Here is such a case: one writes a very long and involved paper that (A) begins with a refutation of an article in a rival journal, and then (B) uses the refutation as a springboard for a novel positive argument. Suppose, furthermore, that step-(A) is critical to the new paper--that is, that due to its dialectical structure (i.e. it aims to use problems with the earlier argument to motivate a novel positive argument), the paper basically has to begin with a refutation of an article in a rival journal. I have some experience with papers like these being desk-rejected "because we do not consider replies to articles in other journals." Further, although in cases like this one may attempt to recast the paper in a different way so that it "appears less like a reply", in my experience another problem can arise here: recasting the paper in this way can actually undermine the paper's dialectical effectiveness (since the primary motivation for the paper's novel, positive argument may be to correct what one takes to be a very big mistake in another person's argument/theoretical framework, i.e. the argument being "replied" to). And indeed, my experience is that the disincentive problem arises here yet again. For although I myself have written a few papers like this over the years (i.e. full-length papers critiquing someone else's framework, and then attempting to provide a better framework of my own), I have mostly abandoned them, preferring once again to write and publish "stand-alone" papers. But this is just to say, once again, that I have been disincentivized from doing something that the citation statistics outlined above indicate: I have been disincentivized from actually engaging with the work of my fellow philosophers. And once again, I suspect I am not alone.

4. The Chicken-and-Egg/'Foot in the Door' Problem

Finally, to return to an issue we discussed last week, I think there are plausible reasons to worry whether the current rule against replies to articles in other journals might make it more difficult for early-career philosophers to "get their foot in the door" in terms of getting their work discussed in the literature. As this study, "Does Science Advance One Funeral at a Time?" suggests, there appears to be a good deal of preexisting "inertia" in the scientific literature. Academic literatures generally seem to revolve around the ideas and arguments of people who already "have their foot in the door." Once a person's work receives "uptake" in an academic literature, discussion of their ideas may generate some momentum, or further discussion in the literature. Conversely, as 'Perplexed Marketeer' noted here, when a person's work hasn't had "uptake" yet, one may have to convince reviewers that the work in question warrants discussion. But, of course, that may be difficult to do! All of which brings us back to the false-negative and disincentive problems. If replies are already disincentivized, and it can be difficult to get reviewers to publish discussions of articles that have no "uptake", the very fact that replies can only be submitted to one journal plausibly makes it all the more difficult for an article to get the kind of "uptake" in the first place necessary for reviewers to be amenable to papers discussing one's work. 

So then, or it seems to me, there are several related reasons to worry about whether the current rule against replies to papers in other journals plausibly erects several barriers to philosophical discussion. Given the current status of philosophical discussion today--where, again, 80+% of papers are never cited, let alone discussed--I think these things should worry us. If anything, our discipline should be trying to encourage more cross-discussion, not squelch it. Why, then, do so many journals have such a rule? As far as I can tell, there are two plausible reasons for it:

1. Volume: considering replies to articles in other journals might overwhelm journals with submissions, adding to an already saturated peer-review system with long wait-times and not enough reviewers.
2. Self-interest: as 'Insightful diner' noted here, journals have self-interested reasons against promoting discussion of articles in other journals (viz. increasing another journal's citation-rate/impact factor can undermine the prestige of one's own journal!).

For reasons I explained in response to 'Insightful diner' (and which Steven French seconded), I am unpersuaded by the self-interest concern. First, opening up one's own journal to replies to articles in other journals promises to improve the content of one's own journal. It would plausibly provide readers with refutations and constructive engagement with the rest ofthe literature that (for reasons given above) are currently disincentivized. Second, critiques, refutations, and improvements of work in other journals appearing in your own journal promises to increase the prestige of one's own journal relative to others (as it would show readers that one's own journal publishes stronger work than the work in other journals being critiqued). 

I am much more sympathetic with the volume concern, as indeed I think allowing replies to articles in other journals plausibly would  increase journal submissions substantially. Nevertheless, I am optimistic about whether the volume concern can be surmounted. First, some journals (such as JESP) already consider replies to articles in other journals, and JESP has some of the quickest turnaround times around! Second, it seems to me that journals could handle volume problems creatively--for instance, by merely asking for a "thumbs-up/thumbs-down" verdict from reviewers (as opposed to the standard written justification for reviewer recommendations), or perhaps placing a cap on how many reply submissions they can accept for review annually. Although when it comes to standard articles I am against these approaches--as I think authors generally deserve some justification for why their articles are rejected, etc.--it seems less problematic to me in the case of replies (or, at least, a legitimate tradeoff to accept given the total costs and benefits involved of journals publishing more replies).

In any case, although I think the volume concern is a serious one, I think there are a number of reasons for our discipline to have a more open conversation about whether philosophical engagement might be improved by removing or modifying the standard editorial restrictions on replies, and if so, how journals might do so effectively and feasibly.

Comments

[Steven French]

Maybe we need a new journal: 'Discussion Notes'

(Here's the obv point where the volume issue bites, at least when it comes to journals that still retain a print version - the more replies you publish, the fewer original papers. So, a reply is going to have to be *really* good to bag the spot that would otherwise go to an original contribution ...)

[shane wilkins]

My thought was along the same lines as Steve French's, with the addition of an observation about citation rates.

Other things equal, one would prefer to edit a better journal than a worse one. Since one common metric people use to judge the quality of a journal is citation rates, therefore one has a reason to want to increase the citation rate of one's journal. Further, replies to pieces in other journals are less likely to increase your journal's citation rate than original pieces, so it makes sense to bar replies to pieces in other journals. (Note that journals less commonly bar replies to pieces in their own pages--and the citation rate hypothesis explains why a reply piece automatically guarantees at least one citation to the original paper, and a rebuttal by the original author might generate even more.)

Perhaps that's an overly cynical view, but it seems to me that people's behavior in the profession is often shaped by precisely these kind of quantitative metrics.

[Shane]

It already exists. Check out the online journal American Dialectic: http://www.americandialectic.org

[Marcus Arvan]

Hi Shane: Very cool!

American Dialectic has a very neat format, publishing original articles and then continuously adding published replies below those original articles. I think this is great idea, and hope American Dialectic can rise to greater prominence in virtue of how it generates discussion! (As an aside, I would suggest to the editors of AD that perhaps they might include links to replies below the original titled article on front page of each "volume", below the titles of original papers. Currently, one has to actually click on the original paper to see whether there are any replies! I suspect listing the replies on the first page of the Volume might entice readers more).

At the same time, it is not entirely clear to me that American Dialectic is exactly a "replies only" journal. Rather, as far as I can tell, it publishes original articles which can then be replied to. Although I think this is great, I still think the discipline could use a journal dedicated merely to replies to articles in other journals!

[Tom]

I agree with Marcus. I also think that such a journal is liable to be heavily cited. I see this happening essentially in the following way: suppose you want to use article X as part of an argument. A natural thing to check would be if X had been replied to in the replies journal. If it had, then it would be natural to cite the reply along with the article.

Thus, instead of `See X', the footnote (or whatever) could read `See X. For an interesting response to X, see Y', where Y would be in the replies journal.

[Derek Bowman]

It is not a "replies only" journal, but I'm sure the SERRC (Social Epistemology Review and Response Collective) would be happy to post short replies to journal articles published elsewhere. See https://social-epistemology.com/about/

What do you think the advantages are of a peer-reviewed replies journal over a venue like the SERRC which is a more open platform affiliated with a peer reviewed journal?

[Helen]

For the reasons Marcus outlined, it's tough to get a pure reply piece published. This is probably why I haven't written any replies except short response papers for BBS and other journals that have this format (it used to be that you pitch your reply, and they then decide to invite you to write the full paper, which is nice in terms of time investment). I have, however, started writing replies to papers that I then put in a larger framework, situating my piece as an original contribution to the field, and there is then just a section of that paper critiquing a recent other paper (I recently got a paper into a top-10 general journal that uses this strategy, and I have done it before, so it pays off). I do think that pure replies have value and not every idea lends itself to being placed in a larger frame like that.

[S]

Hi Helen,

Do you have any advice (that you would be comfortable sharing) about how to turn a reply paper into an original contribution. I've got some abandoned reply papers that I put a lot of work and heart into that ended up being rejected by the journals in which the papers to which they were replies appeared. It would be an unexpected delight if it turned out that some of the papers could be turned into something that eventually found a home at a journal.

Thanks!

[Marcus Arvan]

Hi Helen,

Like 'S', I too would be very interested in any advice you have there. I have tried on a number of occasions to convert replies into "original contributions", but in every case so far I have run into trouble with editors treating (and rejecting) the pieces as a "reply to an article in another journal."

Thanks!

[Helen]

Hi S and Marcus: Sorry for the late reply. You can e-mail me for more details. What I did was start the paper off with an introduction that frames the broader debate. Say you want to reply to some influential philosopher's take on a given problem X, then the intro should be something about that people have worked on X, and there's this unresolved issue about X. You can then also introduce the piece you're responding to as one particularly influential view in the debate on X. You then outline your novel contribution to X. It's important to also have a constructive piece and not just the critique, otherwise it will be read as a reply. That's when you put up the reply to the philosopher you're targeting. I'm not sure this strategy works always, but it has worked for me (in a journal explicit about not publishing replies, and the paper ended up, I think, as an original contribution to the field that's larger than a reply - but my very first draft was a pure reply, which I then broadened out. I started out at about 3500 words, and ended up with about 8500. I think it helps that the paper you're responding to is fairly recent (< 5 years old in philosophy, I guess)

[S]

Hi Helen,

Thanks! That is helpful.

S

[Wolfgang John]

For the reasons Marcus outlined, it's tough to get a pure reply piece published. This is probably why I haven't written any replies except short response papers for BBS and other journals that have this format (it used to be that you pitch your reply, and they then decide to invite you to write the full paper, which is nice in terms of time investment). I have, however, started writing replies to papers that I then put in a larger framework, situating my piece as an original contribution to the field, and there is then just a section of that paper critiquing a recent other paper (I recently got a paper into a top-10 general journal that uses this strategy, and I have done it before, so it pays off). I do think that pure replies have value and not every idea lends itself to being placed in a larger frame like that.