Peer-review and sub-disciplinary silos

I'd like to share a vague worry I have about the effects that "sub-disciplinary silos" may have on the peer-review process and philosophy more generally. What is a sub-disciplinary silo? Very roughly: a group of specialists in a specific sub-area of philosophy who primarily read and review work in that area. Here's a personal illustration. I'm increasingly asked to review papers for journals. Because I've published in nonideal theory in political philosophy, as well as on transformative experience, I often get asked to review papers in precisely those areas. I don't often (and in some cases, ever) get asked to review papers outside of my sub-disciplinary specialties--say, in metaphysics, epistemology, or philosophy of language. Generally speaking, I suspect I am not unique in this respect. The peer-review process generally seems to something like this: a person sends a paper to a journal, the editors look for specialists in the area to review it, and decide whether to publish the piece on the basis of the specialists' recommendations.

On the surface, this looks all fine and good. Who is in a better position to evaluate a paper than a specialist in the area in which the paper is written? Alas, I have a concern. In the sciences, papers are tested against the world--against a mind-independent reality. Scientific papers make hypothesis, and then test whether those hypotheses are confirmed or disconfirmed by data. In philosophy, on the other hand, we test papers against each other, making a case to our readers that our premises are true or plausible. In which case it matters who one's readers are. If one's readers are a biased sample--a group of self-selected individuals united, say, by similar philosophical commitments--then even if people in that sample find one's premises true or plausible, that may not actually be good evidence that one's premises are true or plausible. And herein lies my worry: when papers are read and evaluated primarily by specialists in a given sub-discipline, it is possible that the very group in specialists in question may comprise this kind of biased sample.

I came to have this worry in part by some of my experiences at conferences, as well as in part by some experiences as a journal reviewer.

At a number of conferences, I've attended sessions in really narrow sub-specialties: say, sessions on incompatibilism about free will, sessions on compatibilism, and so on. It has struck me in these cases that the audience often does not seem very representative of philosophers. For instance, incompatibilist sessions tend to seem chock full of incompatibilists; compatibilist sessions chock full of compatibilists, and so on--in other words, people who tend to share each other's philosophical commitments and intuitions, even when there are people in other sessions who don't share those things one bit. Indeed, when I'm in, say, a compatibilists session, and compatibilists in the audience nod approvingly to something said, I think to myself, "No incompatibilist would ever accept these premises." Yet there seem to be no incompatibilists in the room who speak up to say, "Wait a minute, I don't accept that!" Conversely, if I attend a session on incompatibilism, I've often found myself hearing arguments that the many incompatibilists in the room seem to find plausible, but where I think to myself, "no compatibilist would ever find these premises plausible."

I've had similar experiences on occasion as a journal reviewer. Every once in a while I get a reviewer assignment somewhat outside of my area of specialization. In some of these cases, I often find authors appealing to premises that are apparently considered "plausible" by people in that specialized area, but which as an outsider don't find very plausible at all. As I sit and read the paper, I think to myself, "How could people in this subdiscipline take this premise seriously?" And I cannot help but wonder whether, if papers in that area were reviewed by outsiders, things might look very different: intuitions that insiders find attractive not accepted by outsiders, for instance.

So that's my worry. My worry is that if papers are primarily sent out to specialists in a given subdiscipline, they may in effect be judged by a biased sample--by people whose commitments and intuitions may not be very representative of the commitments and intuitions of philosophers or inquirers in general. Suppose in Plato's dialogues Socrates walked around systematically avoiding people who didn't share any of his commitments about justice, engaging instead primarily with people who do share them. I hope we can agree that there would be something very odd about this. Far from being a "disinterested search for the truth", Socrates would be searching for things himself and a self-selected set of interlocutors find plausible...even though there might be many other people out there who don't find the same things very plausible.

Is there a way around this problem? Here's one possibility: papers sent out for review should be reviewed by at least one specialist, but also at least one "outsider"--someone who's not steeped in the literature, theories, intuitions, etc., of that sub-specialty, as a kind of a control group against sub-disciplinary silo effects (or bias). 

What do you think? Would this sort of practice be a good idea? Do journal editors already do something like it? I'm curious to hear what you all think!

Comments

[Shen-yi Liao]

I think the main point is plausible, but I think non-experts would be more effective if their judgments are pooled together collectively. However, given that the standard remains one or two referees, I am not sure non-experts do do better.

I do want to object to the contrast between scientific and philosophical papers. Scientists test their papers against each other too: which hypotheses are relevant, which analyses are appropriate, etc. Scientists, as agents in academia, are also impacted by sociological factors such as career incentives. It doesn't seem obvious to me scientific peer review is any more "objective" than philosophy peer review, if that's what the contrast is supposed to imply.

[Amanda]

Well, another bad thing is that this is the way I would guess that 1/2 of journal papers are not really blind review. The circle is so small you know who wrote the paper, whether via conferences or just paper sharing.

[Ryan]

"In philosophy...it matters who one's readers are. If one's readers are a biased sample--a group of self-selected individuals united, say, by similar philosophical commitments--then even if people in that sample find one's premises true or plausible, that may not actually be good evidence that one's premises are true or plausible."

This is another reason why a referee who rejects a paper at one journal should not agree to review it again at a new journal. Let some air in.

[Marcus Arvan]

Ryan: totally agree!

[Marcus Arvan]

Amanda: good point!

[Marcus Arvan]

Shen-yi: I agree that we shouldn’t exaggerate the difference between science and philosophy. But there is a difference: there is clear mind-independent data in science. For instance, people could (and did historically) protest the counterintuitiveness of relativity, quantum mechanics, or evolution by natural selection all they like. The data show otherwise. The problem is, in philosophy, there often isn’t anything remotely like mind-independent data: there’s *just* different philosophers’ judgments about which premises are plausible, implausible, etc.