Stanley on peer-review

Jason Stanley has shared his experiences with peer-review on facebook, and consented to have them shared here. Stanley writes,

I see a lot of junior faculty stressed out about getting their papers published in peer review journals. I'm posting this in the hopes that it will help people. I have never had good luck with peer reviewed journals. Around 2/3 of the papers on my CV have been accepted blind peer review, but only four or five of those have been accepted to the first journal to which they have been submitted, almost all of them as R&Rs first (and one of those was a neuroscience journal). My 2002 paper "Modality and What is Said" was finished in 1996 and rejected from 11 journals. And yesterday's desk rejection without comments was the fifth desk rejection without comments I have received since 2011. Here is my view about all of this, after 20+ years being in the mix. There is a lot of sociology to peer review. Whether you happen to be working in an area filled with schadenfreude and resentment, or where people are happy and mutually supportive, makes an immense difference. Philosophy is just a very small field and resentment and pettiness are features of some areas, and mutual support features of others, and it's just luck which you end up working in. Fortunately, in the end, in philosophy, whether your papers are accepted to leading peer reviewed journals doesn't much matter, even for tenure, as long as they come out somewhere. In my experience, papers take on a life of their own. Four of my papers that were rejected from multiple top journals subsequently became among the 20 most cited papers in those very journals since 2000 (of course citation is problematically political and messed up in the very same ways that peer review is; I'm just saying that there are multiple ways to gain recognition in the field. Each is messed up but you only need one route). Also, a tenure file is a bunch of papers that are carefully read. I have seen people not get tenure because of bad papers published in top journals, and I have seen many more people get tenure because of fantastic papers published in supposedly lesser journals. There is a silver lining to getting a paper rejected from a journal; I am sitting on five papers now that didn't get into journals, and some of them of are really awesome. That's many years between me and having no bullets in my gun (always a fear, perhaps an irrational one, but a fear nevertheless, and it's good not to have it). In the end, the best advice I know I got from a brief conversation with Robert Nozick. He told me when he sent a paper out to a journal, he would first prepare a stack of envelopes, addressed to different journals. When the rejections came in, he would simply slip the paper into the next envelope.

Any thoughts? One thought that initially comes to mind is a bit of relief. It's (somewhat) reassuring that even very established scholars in the field (such as Stanley) face systematic rejection, including desk-rejection. On the other hand, it is also disconcerting! If, as Stanley writes, some of the most cited papers ever in top-ranked journals have been systematically rejected (in some cases, desk-rejected) by journals in that same echelon--in some cases, over ten times--what does this say about reviewing standards? Following our recent discussion on whether it is "too hard" to publish ambitious/groundbreaking/paradigm-shifting work in journals, one might wonder whether Jason's experiences are (another?) data-point in favor of the notion that standards might be "off" in some way.

I don't know for sure, but in any case, Stanley's experiences struck me as worth sharing, if only to reassure readers that, no, we're not the only ones who face the "indignities" of peer-review. :)

Comments

[Wesley Buckwalter]

Well said, this also reflects some of my experiences, insofar as reports are often unactionable, unhelpful, or end up failing to track publishable quality. To register one small point of disagreement though, I'd be pretty surprised to learn prestige publications were incidental to influencing tenure promotion deliberations, though I have no experience serving on those committees so don't really know.

[Derek Bowman]

This is small comfort for those of us on the academic job market trying to find a way to stand out among dozens upon dozens of other qualified applicants.

The advice Stanley got from Nozick (I've heard similar advice that others have attributed to Derek Parfit) may represent a good strategy for many individual philosophers. But think of all the professional editorial and review efforts that go into each of those submissions, as well as the stress that process provides for those who are dependent upon publications for employment or promotion. Is this really a worthwhile use of everyone's time and intellectual energy?

[Ambrose]

"what does this say about reviewing standards?"

I think if we were less biased by our investment in this whole 'profession' with its traditions and norms, etc. it would be plain as day what this kind of story says about the 'profession' as a whole. Peer review is garbage. (To forestall the sneers: this is not (only) sour grapes. I've had positive reviews and papers accepted at 'first-tier' journals.) The profession is morally and intellectually bankrupt. The peer-review process is objective or rational or meritocratic or philosophical only at times, and then only by accident. There is nothing about the system or procedure or norms of the profession that plays any role whatsoever in making this occasional reasonableness or fairness happen. Same goes for admissions to programs, hiring and promotion and tenure decisions, and pretty everything else in the 'profession' (e.g., consensus about which thinkers or problems or paradigms or methods are significant or deserve to be studied and taught).

Those we call 'philosophers' do not actually know anything of any importance that others don't know in virtue of their 'professional' training. They are not, as a group or by training, any more intelligent or insightful or rational or ethical or virtuous than anyone else. There are no real standards that 'philosophers' use to evaluate the work of their 'peers' or for deciding who is a 'peer'). There is just a lot of 'sociology' as Stanley euphemistically says: nepotism and fashion and peer pressure, group think, mindless acceptance of bad ideas mindlessly accepted in the broader culture, cult of personality, etc. Still, philosophy does exist (somewhere else) and we can always try to improve ourselves by studying philosophy. But this activity has no systematic relationship of any kind to what goes on in our 'profession'.

Marcus you're a cautious and thoughtful writer. You want to just raise the question of what Stanley's story says. But at some point, when we take into account the billionth story of this kind, shouldn't we accept what seems so obvious? Especially when, as you have often pointed out, there are so many other, independent reasons for _extreme_ skepticism about the norms and approaches and assumptions that make up academic philosophy? We should reject it (though not the good and true things that have ended up within it, by accident) and consider seriously all of the alternatives to it.

[Postdoc]

I am a recent PhD graduate. I am effectively employed 1/3 time at the moment. My wife works 3/4s time. So, together we have a median income. But it is unknown whether her contract will continue. Thus, rejections don't just hurt my ego. The very quality of my family's life is on the line. This is my situation, and it's a situation many are in. In fact, our situation is much better than some people's.

I understand that journal's aren't charities. But when so much is on the line, it is unacceptable for the peer review process to be as poor as it is. The process should more or less work. It should more or less fulfil the moral obligations to authors and to the discipline. Reviewers should not take 6-12 months to review an article. Editors should not use incompetent or overly hostile reviews as an excuse to keep the journal's rejection rate at 95%. Reviewers should not backtrack on their own opinions. Journals and referees may not be charities, but they do have some moral responsibilities to the author and to the discipline. Unfortunately, the peer review process does not work to fulfil its obligations. At best, as Ambrose says, it works by accident. It does not consistently work or even 51% of the time.

Although I have had good experiences and successes publishing in top philosophy journals, the horror stories I have would blow your mind if they weren't just normal. As philosophers, we've come to accept some of the most atrocious referee and editorial behaviors. I've had totally incompetent referees who were obviously, repeat obviously, completely ignorant of the field in which I was writing. I've had referees change their minds 3 times from minor revisions, to R&R, to reject, with the final reasons given for rejection just being incoherent nonsense like 'he was too eager to make the changes I asked for.' Seriously, a referee told me that once. I've waited half a year for a 1 page report where the referee only read the first section of the paper. I've had the same incompetent referee reject my paper from multiple different journals, and none of the editors ask why he is rejecting the paper (to see whether he even has sensible reasons). I could go on and on.

As authors we've habituated to this horrible treatment and find various ways to cope. From an outside perspective, or if you step back and reflect, it should be obvious that philosophy is basically a dead profession. There are no jobs but for the Ivy league or the well connected. Everyone else spends huge amounts of unpaid time trying to contribute to the profession by fighting a peer review process that is just horrifically bad. In the end, many will drop out of philosophy in their 30s and struggle to pull themselves together and find a new career. If they manage to be good caring people after years of struggling with the peer review process and the philosophy job market it'll only be by an amazing act of will.

[Derek Bowman]

Don't worry, Postdoc. Plenty of Ivy graduates are struggling for jobs too.