Persistence in peer-review

A while back, I shared some of Jason Stanley's thoughts and personal experiences with peer-review:

I have never had good luck with peer reviewed journals. Around 2/3 of the papers on my CV have been accepted blind peer review, but only four or five of those have been accepted to the first journal to which they have been submitted, almost all of them as R&Rs first (and one of those was a neuroscience journal). My 2002 paper "Modality and What is Said" was finished in 1996 and rejected from 11 journals. And yesterday's desk rejection without comments was the fifth desk rejection without comments I have received since 2011...

The other day, I came across this Twitter thread featuring a few other philosophers sharing screenshots of how often their papers have been rejected. Here's one from Wesley Buckwalter (Pittsburgh):

And here's one from Michael Cholbi (CAL Polytechnic):

Good times. 

Comments

[Pendaran]

Yes, this is my experience too. Of my dozen or so papers most were rejected a ton before eventually receiving an R&R and then being accepted. In my experience an R&R almost guarantees acceptance (80-90% probability). However, it's very hard to get an R&R. The majority of journals and referees mainly look for any excuse they can to reject. It's just the nature of the game I suppose. Way too many submissions! The profession is utterly flooded with fresh PhDs for which jobs do not exist. The publishing war that is thus ensuing has made publishing a worse experience than it probably ever has been.

[Al]

Not only have I had a similar experience, but I wish I had been given some advice: just because a paper didn’t get into a journal deemed very good, you don’t have to ‘go down’ the chain. I’m not advocating that the profession act this way, but where one publishes does matter (for getting a job and for getting read) and early on I thought a rejection from a well known generalist journal meant I should try for a less prestigious journal. I later gave this up. To give one example, it took 7 tries but a paper landed at Nous.

I did get some advice/words that helped both practically and psychologically: an accepted paper is a paper two people liked. But I wish someone told me sooner and in clearer terms: send your work to places that seem like a good fit for the length, scope, and topic, but shoot high and stay high until you can’t any more.

[Amanda]

One thing that is funny about all this, is it seems to me this should seriously make us rethink the peer review process. And yet we keep doing it and acting as though it is a clear measure of merit. Perhaps more than anything it is a clear measure of persistence, and the ability to write A LOT (as then you will have a lot of papers out, and odds of acceptance are higher.) IDK, it just really bothers me that my papers that are published, and some in quite prestigious journals, are ones I have no reason to believe are better than my other papers that are not published. Indeed, I would say with some confidence that my published papers are, on average, no better than my rejected papers. And Jason Stanley's story confirms this is true of others too. But if published papers are no better on average than unpublished ones (at least when the comparison is papers of the same person) well, that is kind of depressing.

[Pendaran]

The peer review system was designed a long time ago when competition was minimal compared to today. A PhD from a respectable place used to almost guarantee you an academic job, and few would have more than one publication when getting their first TT job. People applying for 1 year teaching posts often now have CVs that would have been sufficient for tenure 20 years ago, even 10 years ago. The entire system needs to be redesigned to take into account the publishing war currently ensuing, but it won't be. Everyone is too disorganized and overworked. The best we can hope for is that honesty about the non-existence of jobs for PhDs will result in reduced demand.
Despite the fact that our peer review system is a disaster, I don't think that published papers aren't a clear measure of merit. We're told that we need to publish. Those that do better at this worked really hard at it. Hard work should be rewarded. I worked my butt off, almost to the break of sanity, amassing 11 papers in 3 years. Also, the peer review system is certainly a filter of quality. Persistence is a giant part of publishing success but quality is relevant too. Most published papers I have read are written fairly well and have interesting and novel, even if unconvincing, arguments.

[Amanda]

I guess my experience reading journal articles is different than yours, Pendaran. Or we just have different tastes. I will say that most published papers show competence in analytic reasoning, intelligence, and the ability to write well. But to me this does not distinguish one philosopher from another. Most philosophers can do this. One thing I would definitely not say about most papers I read is that they are interesting and novel. Indeed I would say exactly the opposite. Most (but of course not all) papers I read in journals I find boring and derivative. Indeed, some of my own papers I find most interesting are ones that have not been published, while some of my more boring papers have!

Publishing is some measure of merit. And when compared to the other measures, it is one of the better ones. But I still think it is not that good most of the time, and the fact that some of the most well respect papers in philosophy couldn't make it past the peer review system (or they took 15 times to get there) speaks to how deep a problem it is.

[Pendaran]

"One thing I would definitely not say about most papers I read is that they are interesting and novel. Indeed I would say exactly the opposite. Most (but of course not all) papers I read in journals I find boring and derivative."

Maybe this is a function of our research areas. I find most of the papers I read to be interesting and novel, even if incorrect and even obviously so. I guess my complaint about philosophy is different from yours. I think most of it is silly.

Modal realism is a great example. It's silly, but we take it seriously. Does that mean it's not interesting and novel? No, not necessarily. It's certainly a "cool" and profound idea, which many will find interesting and novel. But it's not a serious attempt to get at the truth of modality.

The situation is analogous to sci-fi vs. science. A lot of philosophy I'd place into the category philosophy fiction. Many papers attempt to tell philosophical stories but have little regard for the truth. Honestly, I say a lot of science falls into this same category, but that's for another day.

"Publishing is some measure of merit. And when compared to the other measures, it is one of the better ones. But I still think it is not that good most of the time, and the fact that some of the most well respect papers in philosophy couldn't make it past the peer review system (or they took 15 times to get there) speaks to how deep a problem it is."

If what we're supposed to do for tenure, for our jobs, is to publish in philosophy journals, then publishing in philosophy journals is a sign of merit. You might think that philosophy as a discipline sucks. If so, then we as a discipline need to change our standards, make new journals, or revamp the peer review system. You might not like the game, but if someone is playing by the rules, they are playing the game correctly and deserve to win if they play it better.

Also, just because a few famous papers took a long time to get published doesn't mean that's the norm. Further, ground breaking, truly novel work often takes a long time to get published. This is true in every discipline. This is because truly ground breaking research calls into question current established theories and paradigms and wont be taken seriously by most.

[Amanda]

"You might think that philosophy as a discipline sucks. If so, then we as a discipline need to change our standards, make new journals, or revamp the peer review system. You might not like the game, but if someone is playing by the rules, they are playing the game correctly and deserve to win if they play it better."

I wouldn't say the discipline sucks. I would say we need to change our standards, make new journals, and revamp the peer review system. And sure, people who play better deserve to win. But publishing is one way of playing better, and there are other measures. And quality matters as well as quantity. Trust me, I believe many people who are hired are not hired on merit. I don't dispute that.

And I don't have enough data to prove this, but my general idea from everything I have seen and heard, is that it is very common for the papers of top philosophers, or top papers, to take a very long time to get published, if at all. Actually more than a few tenured professors I know only publish what would be journal articles in edited books because they don't want to deal with the peer review process.

[Pendaran Roberts]

I hear you Amanda. I spent years messing with the peer review process. It’s horrendous. I also think my best papers have not been published. However, I just don’t want to overdue the critique and say the peer review process does little or nothing to filter based on quality. That’s not my experience. As I worked my butt off improving papers they were eventually accepted. Many that were originally rejected deserved to be. Of course, many rejected papers were rejected for nonsense reasons, but they all got better over time as I worked on them. So, I wouldn’t say the system isn’t a clear measure of merit. It’s just a very slow and inefficient process that’s extremely stressful and even dehumanizing. I hated it honestly. Who doesn’t?