Some thoughts on gatekeeping

Gatekeeping has a bad rap. Still, though we loathe it, academics do it all the time. Our institutions depend on the well functioning of various gatekeeping mechanisms--mechanisms that help discern what should be funded or published, and what doesn't.

For instance, I am associate editor for journals Ergo and for Journal of Analytic Theology. In addition, I review for journals and for academic book presses (e.g., Cambridge, Oxford, MIT, Routledge, and Mohrbeck Sie, where I am in the editorial board). I have been, and am currently, a reviewer for several grant agencies including the National Endowment for the Humanities and the British Academy. I help decision small grants and prizes for the APA Public Philosophy committee. Also, I regularly review the research of academics who come up for tenure and promotion. Until last year, I was on a committee for grad admissions at SLU, and presently I am on two search committees for jobs searches. 

So, a fair amount of my time is spent evaluating other people's work and trying to discern its quality. But how do we go about being gatekeepers? I think, inevitably, we end up being perhaps a little more conservative than we like, but we do have the power to shape the discipline. How the field advances depends on the quiet, often uncompensated labor of referees, editors, and other evaluators. 

When I got my first tenure and promotion file to review, I wondered how to adjudicate these and how to communicate to the committee if (fortunately  not the case for my first one) if I felt the file was a little thin, or if it plainly did not meet the criteria set out by the institution. I spoke about it at a conference over dinner to someone who has been an evaluator for many tenure and committee reviews. 

"Some people," this professor said over wine and quiche, "Feel somewhat squeamish about giving a negative evaluation. They'll resort to saying something bland or they try to talk up what they believe is insufficient."

"So," I asked "How about you?"

"Well I am not squeamish," he replied "I am part of academia, and tenure and promotion is part of that institution. I value tenure and the academic freedom it affords, but tenure depends on the procedure of the tenure and promotion process. While in most cases, the body of research presents a clear case in favor of tenure, it sometimes doesn't, and the committee needs to know that. I need to be honest and diligent in my assessments."

I often think back of this reply and how it can be generalized. When people violate this implicit norm/social contract, problems might arise.

For instance, for one of the journals I am associate editor at, I received a referee report which read (in its entirety) something like this: 

"This is an important, highly original paper. I have no further suggestions. Accept as-is". 

Based on my own (non-specialist) reading I had some doubts and consulted a senior editor who said "Oh [name] always writes reports like this. You cannot trust this referee. Always just one line that is very positive without detail." Sure enough, a second reviewer whose report came in later identified many problems in the paper and we rejected it on that basis. I don't know what motivates that particular referee. Maybe he believes all research deserves a place for publication. But, working within the procedures and institutions, he delays the process and makes the decision more difficult. Hence, there is some expectation that you follow the disciplinary norms of evaluation.

However, there is a fine line between disciplinary norms of evaluation and the perpetuation of excluding some ideas as "less philosophical" (e.g., feminist philosophy, less commonly taught philosophical traditions), and even questioning whether something is philosophy at all. For this reason, I do think within the bounds of what's possible we should let our own preferences and judgments, subjective though they are, play a role in our evaluations. We're not beholden to how other people would do so. 

To take but one example, I spoke to a philosopher a while ago who admitted a paper she refereed was so boring it was hard to reach the end. She found the conclusion--when finally reached--trivial and uninteresting. Boring and derivative philosophy has its place in advancing the discipline (as J Dmitri Gallow recently argued here), but surely if you find a paper so soporific you can hardly reach the end, that's a ground to reject it? 

"No, I recommended conditional acceptance," she said, to my great surprise. "After all, it's exactly the sort of paper you see in top philosophy journals all the time. I think they're boring and I hardly read "general" philosophy journals anymore. But the discipline seems to think this stuff is great. It was also referee-proofed through and through, so nothing really objectionable. I didn't learn anything of interest, I didn't gain new insight, but well, at least it didn't seem obviously wrong."

I am curious what readers think, but my approach isn't like this. When I, as a reviewer, think a paper is trivial, even referee-proofed to the max, I'll recommend rejection. I am a bit more forgiving of papers that are bold and interesting, or in underrepresented traditions, though obviously they also need to meet standards, it's just that referee-proofing is not a feature that is prominent in my decision making.  If everyone just replicates existing practices, then general philosophy journals will continue to churn out work I'm mostly uninterested in--since what has appeared has changed, and is changing, over time it's possible to tweak a bit at the margins. 

How much you can tweak depends on the position you're in. As a referee you can perhaps help usher a paper through the process that otherwise would never have seen the light of day that you deem very promising and novel. As an editor in chief you can clearly signal that you want change and that you're open to new approaches, as Alexus McLeod recently did for the Philosophical Forum where he explicitly calls for more papers in less commonly taught traditions (one reason for why Johan and I decided to send our paper that combines genealogical approaches in continental philosophy with Indigenous philosophy to that journal). When an editor explicitly discusses policy and openness to newer and alternative approaches, one feels a little more confident that editor would seek out and do their best to find competent reviewers, not simply desk reject a paper in an underrepresented philosophical tradition. 

We may question whether our institutions still work. For example, Marcus Arvan, Liam Bright and Remco Heessen have examined whether pooled/crowdsourced peer review might be better than our traditional system. And perhaps respecting the institution of tenure, given the decline in tenure track positions especially with respect to candidates on the job market, may feel like rearranging deck chairs on the Titanic. But my sense is that once you step into the gatekeeping mechanisms (as evaluator for grant agencies, referee, etc) you cannot suddenly change the system. We need to have collective efforts across the discipline to achieve this.

It's true that individual initiative can shift the field a bit. For example, a lot of my time evaluating other people's work and doing other service is tied up with open access initiatives: Ergo, Journal of Analytic Theology, and Faith and Philosophy (I am managing editor and oversee the publication process, and examine reprint permissions etc). As a result of this, de facto, I have a lot less time to devote to for-profit publishers such as Springer and Wiley. I still referee for these journals, and still submit to them. But because I've prioritized open access initiatives in service,  there is inevitably less time to spend on these. As a referee and editor I encourage work I think is valuable and that I believe is underrepresented. This, for instance, led to two prize competitions I got grant funding for, in feminist and less commonly taught philosophy of religion. 

Of course, not everyone might agree with me that open access (not for profit open access that requires high fees) and less commonly taught philosophy need to be promoted. But, it's inevitable that if you are in gatekeeping positions you have to make some calls about what you believe is valuable, and hopefully a diversity of perspectives leads to good outcomes. This can lead to slow shifts over time, and need to be supplemented by discipline-wide conversations if we want to make bigger changes. 

 

Comments

[wondering]

Perhaps a noob question: What is it to “referee-proof” one’s paper? Just hedge hedge hedge, anticipate anticipate anticipate?

[Anti-Sontagist but not Anti-Sontag]

wondering,

Yes. Another tactic is anticipating and accommodating the views of referees, based on the views that are currently dominant in the sub-field.

So, if Sontagism is the dominant view and you have identified a problem for Sontagism, you don't call it a "problem" but a "point for further research". If you are arguing for a view that implies not-Sontagism, you argue how Sontagism still retains significant value if your view is correct, e.g. as a simpler approximation or as a pragmatically superior view.

This tactic is especially useful given that a surprising number of journals will make Susan Sontag your referee if you are writing anything in relation to Sontagism. Or you might get one of her grad students (even current grad students) as Reviewer 2, who happens to be writing their thesis on applying Sontagism to the free will debate and who will not be amused by a fundamental critique of the whole theory.

As annoying as this form of referee-proofing is, I have actually found that it's improved several of my papers. Often I think I have found a refutation of Sontagism, but it's really a case where Sontag just never got around to filling in some gap in her theory. Often there really is a place for Sontagism as some sort of heuristic, or at least I haven't refuted that. If you have a marginal view and you want to publish in top journals, you have to be more aware of these finer details than someone with a more mainstream view, like Sontagism. This is even more the case if you are not very thoroughly networked in the subfield. It's a blessing and a curse.

[Sam Duncan]

Referee proofing is a couple of things: 1. Citing every single thing written on a topic no matter how blatantly wrongheaded, trivial, dishonest, or just plain *****y you find it and avoiding any criticisms of those works so far as possible. The hope is that referee Jones won't reject your paper out of hand because you fail to cite or sufficiently appreciate Jones's "seminal" and "path breaking" work published in the Zemblan Digest of Theosophy and Deep Thinking. 2. Spending a lot of time and ink addressing every possible objection even those that rest on obvious misreadings of the paper.
It results in papers that are bloated and a pain to read. If there is a good idea it gets buried under the author engaging with enormous amounts of dreck that isn't worth engaging with or, in the case of some wrongheaded objections, doesn't merit the pages and pages the author devotes to it. It's a lot of the reason that academic writing, especially in philosophy, tends to be awful I don't think it's worth doing myself and have stopped even trying to do much referee proofing, especially of type 1. I've decided that if an unfair referee wants to reject you he'll find a reason. Unless you gush about how Jones is right and has solved all problems in this area that type of egomaniac is still going to be annoyed and find some reason to reject. And if you agree with him, well then what's the point of you writing a paper? Beyond that I don't feel that I need to amplify bad work by making readers aware of it. If I think a paper is trivial, shoddy or even blatantly dishonest then I'm not making academia or the world a better place if I talk about it and make sure other people have to waste their time and effort thinking and talking about it too.