The problem of never-responded-to work in philosophy

Over the weekend, a number of my philosopher-friends shared the article, "Why Professors Are Writing Crap That Nobody Reads", on social media. The long and short of the article is that academic publishing has mostly turned into a scoring-system for tenure and promotion:

Professors usually spend about 3-6 months (sometimes longer) researching and writing a 25-page article to submit an article to an academic journal. And most experience a twinge of excitement when, months later, they open a letter informing them that their article has been accepted for publication, and will therefore be read by…

… an average of ten people.

 Yes, you read that correctly. The numbers reported by recent studies are pretty bleak:

- 82 percent of articles published in the humanities are not even cited once.

- Of those articles that are cited, only 20 percent have actually been read.

- Half of academic papers are never read by anyone other than their authors, peer reviewers, and journal editors.

So what’s the reason for this madness? Why does the world continue to be subjected to just under 2 million academic journal articles each year?

Well, the main reason is money and job-security. The goal of all professors is to get tenure, and right now, tenure continues to be awarded tenure based in part on how many peer-reviewed publications they have. Tenure committees treat these publications as evidence that the professor is able to conduct mature research.

A lot of my philosopher friends commented that this is disturbing--but I'm not sure it has been adequately appreciated just why it should disturb us. The real problem isn't that it has turned academic publishing into a cynical and depressing mechanism for personal advancement. The problem is that it undermines the epistemic credentials of our discipline. Allow me to explain.

Consider the following thought-experiment. Suppose that researchers in physics--say, people who work at the Large Hadron Collider--published empirical findings in academic journals year after year. Suppose then that because competition for journal space is incredibly competitive, only some of these findings found their way into "top journals" in the field. Finally, suppose that, moving forward, 80+% of all of the findings were systematically ignored, and never discussed or evaluated in the literature. Suppose, next, that some of the 80% of findings that are ignored contradicted the findings in the 10-20% of the literature that is discussed, providing alternative findings. Finally, suppose that the field itself--including its history books--only reported the 10%-20% of findings that were discussed as "progress in the field." There are fairly obvious reasons to think that if this were the case in physics, it would not be a field in good epistemic standing. The scientific method only works to the extent that findings and hypotheses are attended to and tested, not summarily ignored on the basis of personal or group judgments. If some "findings" are discussed but others are summarily ignored, a serious question arises of whether the "progress" the field reports is more a matter of sociology than sound science or inquiry. 

Might philosophy's epistemic credentials be similarly imperiled? I have suggested before that there are some reasons to wonder whether this may indeed be the case, as sociological forces can plausibly affect the subfields of philosophy that people go into, the arguments they make, which arguments "win out", and so on. But instead of making that case again, I'd like to address a common reply I've come across on more than a few occasions: the reply that most articles in philosophy are "crap" that don't warrant any reply in the literature. Yes, unfortunately, I have heard this more than a few times, from more than a few people. But here's the problem. Aside from being in my view offensively dismissive (we owe our colleagues better than simply dismissing their work), there is a deeper problem with this reply, which I will call the "Trust-Me" Model of philosophical progress and evaluation.

Go back for a moment to the thought-experiment above. Suppose once again that, in physics, 80+% of articles reporting new findings were summarily ignored. Then suppose that a common reply in the discipline were, "Most of those new findings are crap"...despite the fact that almost no one ever actually engaged with the work in question or showed that it is "crap." Here is the basic epistemic problem with this. Good science, and good epistemic practice, is not based on principles like, "Trust me, I know X", or even "Trust us, we know X." Sound epistemic practice involves carefully testing and evaluating hypotheses. It may well be that only 10-20% of philosophy articles are any good. But, if we are to be an epistemically responsible discipline, this cannot be simply asserted on the basis of "personal judgment"--for human beings are notoriously biased creatures. One possibility that article X did not get published in Phil Review is that it is a bad article. Another possibility is that it is a good article and important contribution but some other reason led it to end up in a lower-ranked journal (perhaps the person needed publications to get a job, or tenure, etc.). In order to know which of these (or other hypotheses) are true, the work in question must be tested: its merits and deficiencies must be examined. And again, a critical part of scientific/epistemically responsible practice is that a suitable examination is not merely, "Trust me, it doesn't deserve discussion." We wouldn't accept this model in physics or biology or psychology, and for good reason. "Trust me" is not an argument. If a work is not worthy of discussion, it should be fairly easy to publish something showing why--and we should not assume that it is not worth discussion until someone actually shows it.

If this is right, philosophical norms should change. I've written before about how journal restrictions on "reply" pieces is a problematic barrier to philosophical discussion. I want to suggest that this is only a small part of the problem. Another big part of the problem is disciplinary norms: we don't currently expect ourselves or others to engage with and evaluate with work in a broad range of venues. Currently, academic philosophy consists largely in each person publishing free-standing articles, one after another--most of which, again, are summarily ignored. Replies are the exception, not the rule. And the result is not so much a philosophical discussion--an open exchange and testing of philosophical ideas and argument--as it is a series of unanswered monologues in which only a select few monologues are engaged with according to a "trust me" model of philosophical evaluation and significance.

This is not the way things have to be, and not, I think, how they should be. In some other fields, such as physics and psychology, articles in lower-ranked, out of the way journals are regularly engaged with. For instance, my most-cited paper by far (with 25 citations since 2013) is a psychological study in an interdisciplinary journal. Despite not appearing in anything like a top-journal in the field, academic psychologists read and engaged with the work--because those are the norms of their discipline. They do not work with a "trust-me" model of evaluation and engagement. Psychologists expect themselves and each other to engage with new work wherever it appears. We philosophers can--and, I believe, should--do the same. If we want to be an epistemically defensible discipline, we should not appeal to the Trust-Me Model of philosophical evaluation. We should engage with published work in general, and demonstrate the merits and deficiencies of that work.

How might this be done? Let me float a few possibilities. First, journals might reduce the number of pages for standalone articles, and increase the number of pages for reply pieces, including replies to pieces in other journals. This might not only make publishing standalone articles more competitive, helping to ensure good quality of those pieces; it would plausibly increase incentives for philosophical discussion. Second, we must develop a culture and norms for engaging with each other's work, including work in lower-ranked journals. This might be done by reviewers and editors imposing more stringent controls on citation and discussion practices within standalone articles. If, for instance, a new standalone piece is submitted to a journal (including a high-ranking journal), but that piece only cites and discusses articles in the area from a few high-ranking journals, ignoring relevant publications from other journals, then the piece should not be accepted. It should, at the very least, be given a revise-and-resubmit with suggestions or requirements to cite and discuss relevant pieces in other journals. While I expect this might sound onerous to some readers, it is important to recognize that this is standard research practice in other fields. In fields like psychology and physics, one is not allowed by journals and editors to only cite and discuss articles by "famous people" in "top-journals." One is expected to know and engage with relevant recent literature in general, wherever it appears.

I propose that these kinds of changes in practice would not only plausibly make philosophy a more enjoyable and inclusive place to do work (as it can surely be dispiriting to defend arguments that people just ignore), but perhaps more importantly a more epistemically defensible discipline. If we are after philosophical truth and telling good arguments from bad--as I hope we are--we should not utilize a "trust me" or "trust us" model of evaluation. We should engage with philosophical research broadly and publicly demonstrate the merits and deficiencies of that work. Or so say I. What say you?

Comments

[Eugene]

Great post, Marcus. I like this suggestion in particular:

"First, journals might reduce the number of pages for standalone articles, and increase the number of pages for reply pieces, including replies to pieces in other journals."

One way to supplement this idea would be for journals to stop requiring (by convention) authors to dedicate a third of every paper to anticipating and discussing every possibly relevant objection. Unless discussing a particular objection allows an author to clarify his or her thesis, or refine it, or whatever, let's let readers/editors decide which objections are worth developing/publishing in response pieces.

This could cut the average journal article down from ~8000 to ~5000 words, and might increase the rate at which papers are actually read.

[put a break on it]

Marcus,
I followed the link to ... on average 10 people:

"If a paper is cited, this does not imply it has actually been read. According to one estimate, only 20 per cent of papers cited have actually been read. We estimate that an average paper in a peer-reviewed journal is read completely by no more than 10 people."

If this were an academic journal, we would have sources cited to support such claims. Why should we believe this? Another estimate that has circulated is that you can count the readers of a journal article by multiplying by 10 the number of people citing it (see Price 1986). So my paper that is cited 140+ times has been read by 1400 people. The rationale is that a number of people may have read the article in their pursuit of relevant information for a current research project, one that has not yet been published or may never be published.

[Marcus Arvan]

put a break on it: The claim that the average paper is only read by ten people may be erroneous. But that is not the relevant claim vis-a-vis the point of the post. The relevant claim is that 80+% of papers are never responded to. Whether those papers make sound arguments or important contributions is never tested in the literature, so their absence of discussion can only be chalked up to some kind of "trust me/trust us" epistemic principle--a principle that due to human bias, I do not think we should be satisfied with.

[Henry Lara]

OK, here is a related question/concern: in a discipline like physics or psychology, if a finding is published, there is a certain assurance that *no matter who* did the study, if they followed sound scientific practices, their findings are true. At the very least, if a study has yet to be replicated, an author can be cautious of relying too much on it, but there is a certain epistemic guarantee stemming from the fact that it's a matter of time before the study gets validated. Now, I don't claim that's how science actually works, merely that a lot of people doing science think that's how it works and it may affect their research and article writing and citing practices.

What about philosophy? Philosophers don't, as a rule, present studies that can be replicated (although some do), but mostly present their ideas, or criticize those of others, via arguments. I don't want to dive into the merits or drawbacks of X-Phi, nor do I want to yield to a naturalized philosophy, such as advocated in this blog by Prof. Arvan (although I suspect I just gave him an opening to further advocate for it!) My point is that without the purported epistemic safety net of the scientific method, philosophers lack a way to distinguish between two well developed but opposite arguments...or a million of them. That's where the trust element comes in. I don't think that trust is an entirely bad thing, but I do agree that its susceptibility to bias, as well as other sociological factors like the pedigree of people and institutions, can effectively diminish the epistemic credentials of the profession, not to mention the enormous injustice done to thousands of hard working people. My question is then, is the "problem of never-responded to work" intrinsic to the way philosophy is (as opposed to merely eternally perpetuated bad practices) due to the methods used in the discipline (questioning, critical discussion, i.e., presenting or critiquing rational arguments), which in turn make it susceptible to the "Trust Me" Model? Because if it is, then we may be grossly underestimating the magnitude of the problem.

[Pendaran Roberts]

We do have a citation problem is philosophy, but I hope my stuff is read by more than a few people. My article 'Turning up the volume on the property view of sound' in Inquiry has about 450 downloads according to TandF. There are another 70 downloads or so on philpapers. How many people who download it actually read it? Hopefully 10%. So, about 50 people have read it. That's not horrible, given that all this stuff is pretty niche.

We're not writing popular press stuff. We're communicating with a relatively small number of people in our fields.

[Sam Duncan]

Marcus,
One larger issue here is the extent to which publishing has become an arms race. At most institutions academics need to publish an absurd amount if they want their tenure decisions to be fairly certain. And for people trying to move to a better job, especially those without Leiterrific pedigrees, publishing is one of the few ways they can better their chances. Add to this, the absurdly bad practices of most journals (which you've documented in great detail here) and everyone below the associate professor level faces huge pressures to follow a through a lot of spaghetti at the wall and hope some sticks strategy of publishing. If you're trying to get tenure at an R1 you probably need around ten papers in the pipeline if you want to get the required 5 in good journals before you're up for review (and 6 or 7 in good journals wouldn't hurt if you want insurance). These days the expectations are only somewhat lower at so called teaching schools. And for those without steady jobs they need to throw everything they have out there now just to better their chances for the job search in a years time. All this means I think that a lot of us put a lot of papers out there, and in our heart of hearts we'd have a hard time defending the need of some of them to exist. How many of us really have good ideas for ten papers at the moment? Even if we do practically no one has the time to develop that many papers unless they're at the start of a two or three year postdoc. So I think the sheer pressure of the hiring and promotion market pushes a lot of us to put papers out there whose basic idea we're not entirely sure about or that are very much underdeveloped even if there is a good idea there. (Before I got a permanent job I was guilty of that, and I probably still would be doing it if my current job had the absurd research expectations most jobs do these days). So I think that there's a lot of bad work out there I don't think it's because it's done by bad philosophers but it is yet another symptom of a profoundly broken academic system. The arms race means that most of us don't have the luxury of carefully developing our ideas and it also means that there's such a huge amount of stuff out there (much of it mediocre) that it's incredibly difficult to figure out what's worth reading and what's not except by looking at the journals so good stuff gets ignored. I think our citation crisis is a symptom of deeper problems with academia, and sadly I have no idea how to fix those problems.

[Slobodan Perovic]

1) What I am worried about is that many papers in philosophy are read but never cited. So "trust me" attitude may really be dishonest b/c at least some people will lift ideas and arguments and never credit the sources. I have very recently heard a very convincing and very disappointing account of such a case involving one of the most cited/prominent philosophers. So the situation may be even more bleak than your insightful post suggests. 2) The idea with response pieces in a journal is great. 3) I was amazed recently how editors and reviewers in science policy journals enforce citing of relevant research pieces. You just have to engage with existing pieces: if you think it's crap say briefly why it's crap, or cite it otherwise at an adequate place. We could introduce a practice of that sort in philosophy.

[overworked]

There are lots of key disanalogies between physics and philosophy, but here are a few big ones:

(1) reading a philosophy paper at the level of depth necessary to properly engage with it in print takes WAY more time than reading it at the level of depth necessary to get a rough idea of its quality,
(2) this reading cannot be farmed out to grad students and/or postdocs, and
(3) it is much harder to know in advance in philosophy than in physics what of other people's work will be relevant to a given project.

Given all these factors, the amount of philosophy being produced, and the amount of time philosophers have to spend doing other things, it's just not feasible to assiduously study every paper that purports to have something to say about a given topic. The "trust me" method is obviously imperfect, and surely denies some papers the audience they deserve, but the epistemic labor has to be distributed somehow and I haven't been given a better way.

Of course, one possible fix is crowdsourcing! If you have concrete examples of awesome papers in low-ranked journals that people should engage with more, why not throw 'em up! It could even be a regular feature of the blog.

(p.s. that article you link doesn't do itself any favors with that snide mention of the religion papers. all those titles look super interesting!)

[Chris Stephens]

There are a number of good suggestions here - but I'd rather see philosophers reform their citation practices and habits more than I would like to see more space devoted to "reply" pieces. As it is, I think too many philosophers write papers of the form "Professor Goofmaker says X, and here's why X is wrong..." (see Dennett on "Higher Order Truths about Chmess"). Instead, if you can provide a general (alternative) solution to whatever problem Professor Goofmaker is answering, the fact that Goofmaker is wrong will simply fall out of the correct answer (say, Z).

[Pendaran]

I noticed that in philosophy our lit reviews require no rigorous process. It's just one person's survey of the lit he deems important. In psychology a lit review must use a systematic and rigorous method for going through all the literature. This way it takes bias out of which literature gets reviewed. My wife was amazed that we don't do this. I think we should. It would help citation rates of little known papers that might in fact be highly relevant and important contributions.

[Philosophy Adjunct]

Developing Sam Duncan's comment above:

I think there is a further dimension to the idea that 'most papers published are crap' beyond blithe dismissal of our `intellectually inferior' collegues. There is a sense in which most (virtually all?) publication is insincere. We publish because our MBA-weilding corporate overloads threaten to turn us into adjuncts if we do not. There is the suspicion, then, that most of what is written is not written because the author is really invested, or really take themselves to have anything very significant to say, but because they have to produce high-impact words. I recall reading on the 'Against Professional Philosophy' blog their receipe for writing an article publishable in top journals (http://againstprofphil.org/philosophical-rigor-as-rigor-mortis-or-how-to-write-a-publishable-paper-without-even-having-to-think/). I have received advice from about how to go about publishing which is depressingly similar in content.

There is, then, a sense that most production is not worth reading because we already have a sense that it was produced just for the sake of being produced. When you combine this with the fact that there is just more and more and more to read all the time it is no surprise that people look for some way to narrow down what they have to focus on. You are absolutely right that the 'trust-me' method is a problematic way of making our workloads manageable. But the problem is not just an epistemic one. It is an institutional and politicial one. This predicament strikes me as a prime example of how the corporatization of intellectual life undermines the epistemic worth of what it purports to be improving.

This leads me to propose another solution, which we should take up as a discipline, and which academia in general should take up. Let's publish less! Let's make publishing and conferencing voluntary! Let's find more accurate measures of what makes an academic `productive' and worthy of tenure, and insist on them. Let's argue that the corporate robots that rule university life are ignorant of the value and meaning of intellectual activity. Let's argue they have a perverted conception of learning, knowledge and research which has no place in university life, and drive them out. If we don't cut the infection out at its source I think it is more than likely that your proposal would just shift the `crappiness' from the articles to the replies, without really improving the epistemic credentials of our discipline, or making the academic life worth living.

[Amanda]

Am I the only one who views this as good news? 10 reads! I have published a number of articles in well-respected places but I really was not hoping for more than 4 or 5:)

And yes, as someone trying to make it into the profession I see publications not as a search for ideas or truth but largely a meal-ticket. I figure what I like about philosophy is getting to chat with my fellow professors, read good philosophy, have intellectual peers and work at a university. I really see publishing as a means to an end. (And why? Because when I know no one cares or will read my work I cannot see it as anything else.)

[Sam Duncan]

Philosophy Adjunct,
Thanks for you comment, but I hope it won't seem ungracious if I take exception to blaming all this on the "corporate university" or "corporatization of intellectual life." I do think that there are some harmful trends in the academy that fit those terms. Perhaps the biggest one being the uncritical obsession with numbers and rankings in and of themselves and the failure to ask whether or not those numbers measure anything of value. But most academic philosophers are as bad or worse than our "corporate overlords" on that score.
However, the term misses a lot and it might even obscure more. To be honest I even sympathize with some of the things people label "corporatization." For instance, those of us who work at state funded schools are public servants and I think that gives the public every right to ask whether they are getting their money's worth from us. (Mind you a lot of proposed ways of measuring that are truly awful, but that's another issue.)
I think the deeper problems here are the obsession with research, contempt for teaching, and the way that academy has become a winner take all system where some superstars do incredibly well but most others struggle to even keep a roof over the heads. Most teaching focused jobs pay very little and have no stability so everyone is desperate to get out of them, and those with TT or otherwise stable jobs are desperate not to fall into the adjunct world. The only way to do this is to rack up as many publications as possible, but as I said that creates pressure to publish even when we don't have good ideas or time to develop the good ideas we do have. If the system prioritized teaching or just gave a living wage to teaching faculty it would remove a lot of that pressure and I think the quantity of mediocre work would fall greatly. Moreover, putting more emphasis on teaching would help the quality of research in other ways. Having to teach keeps me honest in my research I've found. If I can't explain to a smart undergrad who's interested in philosophy why what I'm working on is important I'm inclined to think it probably isn't. If more academics had this check I'm fairly certain it would keep many philosophical debates from coming completely unmoored from reality and falling into an obsession with pseudo-problems and pseudo-solutions.