Should we have different reviewing standards for different types of papers?

One concern that has repeatedly come up recently--in various forums--is whether contemporary academic philosophy focuses too much on "nuts and bolts" (or "small debates"), and not enough on big ideas (i.e. groundbreaking, paradigm shifting ideas). A number of commenters raised this concern in this Leiter thread. Similarly, several journal editors and "big names" in philosophy raised the concern here. Finally, just this weekend a philosopher on my facebook feed held an informal poll asking philosophers whether it is "too hard" to publish groundbreaking/paradigm shifting work, and--while the poll was admittedly informal--respondents answered overwhelmingly (>70%) in the affirmative, agreeing that it is too hard to publish such work.

Given that so many people seem to think this, the question arises: what could/should be done to address the issue? What could be done to make it not "too hard" to publish ambitious work? I want to tentatively suggest that the answer might lie with referees--that perhaps we should utilize different standards as referees depending on the type of paper one is reviewing. Allow me to explain.

The dominant standards we  seem to learn as professional philosophers for what counts as a "publishable article" seem straightforward and reasonable, at least at first glance. They are something like this:

1. Does the paper in question significantly advance a philosophical debate?
2. Is the paper's argument clear?
3. Is the paper's argument deductively valid or inductively cogent?
4. Does the paper establish the truth/plausibility of its premises? 

Generally speaking, it seems as though, if reviewers answer all four questions in the affirmative, they advocate accepting one's paper--but if the answer to one or more is, "no", they advocate rejecting it. I don't mean to say (or imply) that this is true of all reviewers, just that these seem to be the dominant standards reviewers use. And this seems to me to be the case. Typical reviewer reasons for rejecting papers are (A) the paper doesn't advance an existing debate enough, (B) the paper simply isn't clear, (C) the argument contains some clear or subtle logical fallacy(-ies), or (D) the author hasn't convinced the reviewer of the truth/plausibility of the argument's premises.

Now, on the surface, these standards seem philosophically appropriate, as they seem to define what counts as a "good argument" (good arguments advance debates, are clear, valid, and have plausible/true premises). Yet, are they really appropriate standards for all types of work? I am not convinced. I am not convinced in large part because--or so it seems to me--if these standards were applied to most "Great Works" (e.g. Kant's Groundwork, Locke's Second Treatise, etc.), the work in question would almost certainly be rejected. Indeed, there's even an ongoing joke based on this point: namely, that the rejection-letters of the Greats would probably look like something like these. Most Great Works have fairly obvious (and serious!) problems--problems that a zealous journal reviewer might harp on as a reason to reject the work.

For instance, is Kant's Groundwork clear? No, philosophers have been debating the meaning of just about every passage for a few hundreds of years no. Are its arguments valid/cogent? This too has been debated for hundreds of years--and some of Kant's arguments seem clearly invalid (Kant's argument in Groundwork III has been called perhaps "the most beloved flawed argument in the history of philosophy"). Etc.

Which brings me to a basic point about the nature of groundbreaking/paradigm-shifting work: namely, that it seems almost impossible to do such work in a way that conforms to the reviewing standards mentioned above. There's a reason why the groundbreaking work of The Greats is often unclear, messy, and rife with problems. The reason is that doing groundbreaking work is incredibly difficult, and the more difficult something is, the more difficult it is to do it well. If Kant were able to write the Groundwork more clearly, filling out the arguments more rigorously, establishing each and every premise, that would have been awesome. But, or so it seems, he couldn't--and that's okay. The work, for all its apparent flaws, was interesting, insightful, and creative enough to spur philosophical debate for generations.

Accordingly, it seems to me that if we want to make it less difficult to publish more ambitious/groundbreaking work, we cannot simply insist on the same reviewing standards all around. Reviewing standards reasonable for nuts-and-bolts work prize a kind of rigor that seems all but impossible (even for "the Greats") to achieve in particularly ambitious/groundbreaking work.

So, then, should we consider--as reviewers--two sets of reviewing standards: standards which might differ according to the kind of work under consideration? Standards such as,

1. Status quo standards prizing rigor for nuts-and-bolts type work, but
2. Reviewing standards for really ambitious work that allow greater lapses in rigor than the first set of standards, provided work has other virtues (creativity, insight, systematicity, explanatory power, etc.) to a very high degree.

Indeed, I'm not sure what other alternative there might be. If indeed it is "too hard" to publish groundbreaking/paradigm-shifting work these days (as again, most people seem to think), there's only one way to change things: one has to make it easier. And there's only one obvious way to make it easier: reviewers adopting different standards than at present.

In short, the following conditional seems true to me: if it is currently too hard to publish groundbreaking/paradigm-shifting work, then reviewing standards should probably change to make it less hard. Notice that I do not assert this conditional's antecedent, nor purport to derive its consequent. That is, I do not mean to assert that we should have different sets of standards. I leave these issues open for debate and discussion! :)

Comments

[grad]

I just wanted to add that while I do think it's probably much too hard to publish groundbreaking, paradigm-shifting work, I think most people overestimate the extent to which their work qualifies as such. As a result, I would suspect that the responses have some noise.

(Full disclosure: I don't think I have any groundbreaking work worthy of publication.)

[Marcus Arvan]

grad: that's a very nice point. Indeed, it's hard to know how to evaluate whether or to what extent assent to "it's too hard" is justified! What are people calibrating their judgments here against? If it's based merely on estimation that it's too hard to publish their own "groundbreaking" work, it's natural to wonder whether their view that it is too hard is based on overestimating their own work.

However, let me offer a couple of alternative possibilities which (or so I expect) might be behind people's views here.

Possibility#1: people often experience work of their own to be rejected by reviewers merely because that work is judged by reviewers (without any detailed rationale) to be "too ambitious." For what it is worth, this has happened to me on a number of occasions. I'm not saying my work is groundbreaking, but on a number of occasions I have had reviewers say a piece is too ambitious and should be rejected--this despite the fact that the reviewer did not give any substantial critique of the paper's argument.

Possibility#2: people see just how often papers rejected (even desk-rejected) at many top journals end up being recognized later on as groundbreaking. I recall a discussion of such examples occurring a while back at NewAPPS, where it was pointed out that Clark and Chalmers' famous Analysis article on the extended mind was previously rejected at a variety of journals, and similarly how some of Jason Stanley's (?) widely cited work from one of his books was rejected at every journal submitted to.

It is this possibility, I think, that probably most motivates the worry that it is "too hard" to publish groundbreaking work. If work that is genuinely groundbreaking systematically gets rejected by journals (as in the above cases), it is natural to wonder whether something is seriously wrong.

[grad]

Nice point. I suspect that there's some of both 1 and 2 in people's responses--i.e. some people have experience with only 1, some with only 2, and some with both. One thing that would be helpful would be to identify a sample of 20 or so "groundbreaking" papers and compare their publication history to great status-quo papers.

On a different note, I think there are some journals that are working to establish themselves as a place for ambitious philosophy. Phil Imprint comes to mind.

[Karl (pseud)]

I think you are correct by and large. However, we should be careful not to conflate papers and books. I am not sure which is easier to publish (I actually suspect it is books). But there are different standards, in my experience. Kant and Locke may be typical examples of book standards. Books seem to get evaluated, not on details or subtle flaws here and there, but by an overall ability to communicate some point that (usually (poorly) paid) referees think is worth publishing. Whereas papers can be sunk if there is some small flaw. One cannot offer a one-line dismissal of a book that a publisher has asked you to review, as is not uncommon in paper rejections.

[Marcus Arvan]

Hi Karl: That's a fair point. But it also raises an obvious question--namely, what (if anything) justifies such different standards for books and articles.

Would section I of Kant's Groundwork have been any less interesting/groundbreaking as an article, rather than a part of a whole book? Would the central sections of Locke's second Treatise have been any less interesting/groundbreaking as an article? In both cases, I doubt it. But, in that case, why shouldn't they be just as publishable as articles as books?

This is important, I think, for the following reason. Publishing articles in highly ranked journals is often (though not always) considered a *prerequisite* for publishing a book (or, at least, a book with a highly ranked academic press). But, or so I've suggested in some previous posts, nuts-and-bolts thinking and ambitious thinking are plausibly different types of skills. A person can be excellent at nuts-and-bolts thinking, but not so good at ambitious thinking; and conversely, a person can be excellent at ambitious thinking, but non-excellent at nuts-and-bolts thinking. If one has to prove one is an excellent nuts-and-bolts thinker (in journal articles) before one can even attempt to publish ambitious work in a good place, this threatens to potentially suppress a lot of really good, ambitious work.

[Karl (pseud)]

While I am not convinced of this, I suspect that there is a case to be made for different standards for books and articles. I agree that there are important issues at stake here.

First there is, I assume, a different economics of books and articles. So publishers have a different set of incentives. Publishers ultimately make the final decision for books, while, academics, by and large, make the decision for articles.

Second, there is a different cost to the reader in two ways. I can invest an hour and almost no money in reading an article or I can invest three weeks of reading time and $47.99+tax into a book. I will expect something different for my investments. I will be a lot more disappointed by a bad book than a bad article.

This, I suspect, means that publishers need to provide a different book experience than article experience. I want a good, big idea that I can use, out of a book. I want a piece of a debate-puzzle in an article, or at least I'll settle for one. I would bet publishers think that is what people want anyway.

It may, sadly, come down to what publishers can monetize. Can you monetize a refutation to a counterexample to a side point in a debate that less than 100 people in the world participate in? Probably not. But you can bundle it with all your journals and claim you are providing the latest research.

To get back to the main point then, this may explain how some big ideas, that can really be spelled out as 23 page articles in full detail, end up as books. Publishers think they will sell, regardless of its technical quality. Since publishers control publishing, it may stay that way and their reviewing standards will win the day.

So we may, in the end not need reviewers to change their standards, but rather alternative models of academic publishing. While this is easier said than done, with new models of academic publishing available (e.g. Gower's new model: https://gowers.wordpress.com/2015/09/10/discrete-analysis-an-arxiv-overlay-journal/) perhaps the tide can turn

[Marcus Arvan]

Hi Karl: Thanks for your reply! No doubt the incentives are different. I'm also all for alternative, arxiv-like publication models like the one you link to. I've advocated for similar alternatives on this blog before, and hope it is the wave of the future--so thanks for drawing my (and readers') attention to that exciting initiative.

Still, as long as the present system is in place, it strikes me as plausible--given people's experiences and opinions on the matter--that rethinking reviewing standards a bit might open up journals more to the kind of envelope-pushing work that so many people evidently desire (to see more of in journals, not just books!)