The crisis in publishing – how to make it slightly less awful (Guest post by the Editors of Analysis - 3 of 4)

By Stacie Friend, David Liggins, and Lee Walters

Philosophy journals are overwhelmed with submissions. We are all aware of the problems this creates. It makes it harder for authors to get timely verdicts on the papers they submit, it makes it harder for journals to reach those timely verdicts, and it means that many potential referees are overwhelmed with requests for referee reports.

The profession needs a solution to this crisis: a better way of doing things, and a way of switching to it from the way we do things now. We admit that we have no solution. Our goal here is more modest: to make some suggestions for how to make things slightly better.

First, let’s cheer ourselves up with some ways in which philosophy journal article publishing is doing relatively well. In contrast to some other disciplines, philosophy usually uses anonymous review: either double-anonymity (where the author and referee are unaware of each other’s identities) or triple-anonymity (where, in addition, the editors don’t know the author’s identity – this is the system we use at Analysis). (Sam Andrews has compiled some data on philosophy journals’ anonymous review practices: https://pjip.carrd.co/#spread). Outside of philosophy, triple-anonymity is rare: of the four triple-anonymous journals Oxford University Press publishes, three are philosophy journals. (In case you are wondering, the fourth is the Journal of Victorian Culture.) No system is perfect, but we think that anonymous review allows editors to make more objective decisions. Also, some disciplines allow authors to propose possible referees for their journal submission. But this runs the risk of creating mutual appreciation societies. So it’s a good thing it’s not the norm in philosophy.

Now for some suggestions for making things slightly better.

When you are submitting a paper:

Consider whether you really need to submit your paper to a journal. Would it be preferable to publish it in an edited collection, or just put it on your website? If the paper closely resembles your earlier publications, is there sufficient new material to justify publication?

Read the journal’s instructions carefully. For example, the Analysis website states clearly that we do not consider responses to papers in other journals. But we receive many such submissions every year. Frankly, authors who ignore journals’ instructions are creating unnecessary work for their colleagues – and wasting their own time as well.

Make the paper as brief as you can before submitting it. As Brian Weatherson has pointed out (http://www-personal.umich.edu/~weath/lda/article-length-section.html), philosophy papers are getting longer. We are unaware of any evidence that their quality is increasing so as to make up for this!

When choosing where to submit your paper, don’t assume that a journal is only interested in receiving papers like the ones they already publish. The reason they haven’t published papers on your topic may well be that none have been submitted! The Australasian Journal of Philosophy says on its website: ‘The current balance of the Journal’s contents is not prescriptive’ (https://aap.org.au/editorial-policy). We suspect that many other journals could say the same too. For example, there are many areas of philosophy in which Analysis receives few or no submissions. The journal used to have a narrower scope, but since 2016 Analysis has been keen to accept excellent short papers in any area of philosophy.

When you receive request to referee a paper:

Respond promptly, if at all possible. If you can’t referee, it’s usually very straightforward to decline. (And if you are too busy even for that, consider putting an auto-reply on your email account saying something like “Refereeing invitations not responded to within three working days should be taken to be declined”.)

If you can, accept the invitation and write a short report aimed at advising the editor on what verdict to give the paper; when it comes to alleviating the crisis, we think that this is better than accepting refereeing invitations less frequently and writing long reports. We said more about this here.

Accept the invitation even if you have refereed the paper (or an earlier draft of it) previously. Your expert judgement on the paper is still valuable even if it has already been given to another journal. Consider a small subfield of philosophy – one where only a handful of people know the literature in depth. Imagine that each expert in this subfield writes one negative report about the paper then declines any further requests to referee it. If the author keeps submitting it, eventually every expert in the subfield will have written a report on it, which is hardly a good use of philosophers’ time. And eventually journals will have to resort to non-experts to evaluate the paper. That means it runs the risk of being published in spite of its poor quality, and of taking up space that might have been occupied by a paper that really deserved to be published.

When you are invited to revise and resubmit a paper:

Don’t feel you have to adopt every suggestion made by the referees. If you are clear that a modification suggested by a referee would damage the paper, you can explain in your covering letter detailing the changes why you are not adopting it. Often, discussion between the referees and author should happen off stage rather than in the published text of the paper.

Try to resist the temptation to lengthen the paper by adding material along the lines of “An anonymous referee has said this … but in response to that, I say this…”. See if you can deal with the referee’s point by cutting material, or by rewording what is already there, rather than by adding commentary on the referee’s points.

When preparing your CV:

Unless your institution forces you to, do not list the word counts of accepted articles. In philosophy, there is little or no correlation between length and philosophical significance. A paper that is ‘substantial’ in one sense may well not be in another sense. Listing word counts only reinforces the impression that longer papers tend to be better. That has two problematic consequences: it adds to the pressure on journal space, and it adds to the pressure on everyone’s time, because longer papers take longer to read.

Rather than listing the journals one has refereed for, list all the refereeing work one has done. Thus ‘Analysis (2016, 2017, 2018, 2019 x 2, 2020, 2021 x 3, 2022)’ rather than just Analysis. (See https://faculty.philosophy.umd.edu/pcarruthers/ for a CV like this – and an impressive record of refereeing work!) The practice of just listing names of journals means that the second and subsequent reports for the same journal make no difference to the CV. That tempts philosophers to increase the number of journals they have refereed for, rather than to increase the number of reports they have written – something that is not helpful to the profession. Our recommendation makes clearer the contribution that the referee has made, and gives credit where credit is due.

We are under no illusions that these suggestions will solve the crisis. But we think they will make the situation slightly better. And making something slightly better makes it better.

Comments

[Richard Y Chappell]

> "Accept the invitation even if you have refereed the paper (or an earlier draft of it) previously."

I think this is a bad practice, and risks obstructing especially novel or iconoclastic papers that may prove polarizing. Such papers are often very worth publishing, despite only a minority of expert referees initially recognizing this. If you encourage the same gatekeepers to re-apply their conservative negative judgments for each new journal the author applies to, this reduces their chances of finding the fair-minded referee who will recognize the paper's importance.

> "it runs the risk of being published in spite of its poor quality"

I just don't think this matters all that much, compared to the risk of failing to publish really good iconoclastic papers that too many are initially biased against. (But I also think that journals should shift to online-only, so that "taking up space" is not a constraint that prevents them from publishing other papers judged to be sufficiently high in quality.)

[Anonymous]

Could triple-anonymity disproportionately affect disadvantaged groups? Although identity might not bias referees through name, it may come through the writing (e.g., native vs. non-native English writers and gender differences).

[delete or expand]

I understand complaints about length. Still, deleting sections in response to the reviewers, as opposed to expanding to address their comments, often feels more difficult, and like more work. I think this is partly because one can have the sense that doing so disrupts the flow of the argument.

'So I just deleted section 4, but it doesn't seem like I can now transition from section 3 to 5 by just adding a sentence; they're very different. Section 4 was the link! So now what?' This is the kind of thing that I find happens if you try to shorten in light of comments rather than expand, and it involves a lot of intellectual labor to get the paper back into shape.

I suppose if editors want shorter papers, then referees should make sure they are not asking for revision work that will be naturally taken as a request for expansion. That, or we should get more direct comments: 'Section 3 can go and the paper will be fine'.

[Crisis manager]

As the editors suggest, certainly part of the crisis is a function of the massive number of submissions that should not be submitted, but are submitted again and again. Contrary to what has been suggested in other conversations, there really is a law of diminishing returns. More - that is JUST more - publication does not lead anywhere. Indeed, I once heard an excessive early career publisher referred to as a bottom feeder. I was taken aback. But I came to understand what was being said. After you have published a few things, aim higher ... so write better papers. That will serve you well in the long run.

[Bill Vanderburgh]

Has anyone ever done an "inter-rater reliability" study on philosophy refereeing? Something like: have 20 people review the same paper and see if there is any consensus in the details of the praise and criticism it receives (then do that another hundred times or whatever the statistics require). I rather suspect that although referees are often very confident about their judgements, there isn't much uniformity between them.

If I'm right about this, then we are kidding ourselves with low acceptance rates. (As I recall, philosophy has one of the lowest journal acceptance rates of any academic discipline: https://dailynous.com/2018/05/24/insanely-low-acceptance-rates-philosophy-journals/.) And if that is right, there will be no measurable drop in quality if we double the number of papers published. (Tiny differences between papers, unreliably judged, currently make huge differences in outcomes.) This would reduce the refereeing burden, since papers that would have been sent serially to several journals before eventually being published will instead go to just one (thus using two referees instead of four or eight).

I predict this move would also reduce referee report length (less need to find enough reasons for rejection), and thus reduce the corresponding burden of refereeing, making more people willing to referee more.

[grymes]

"Consider whether you really need to submit your paper to a journal. Would it be preferable to publish it in an edited collection, or just put it on your website? If the paper closely resembles your earlier publications, is there sufficient new material to justify publication?"

Material realities aside, this is excellent advice. But who is it advice for? Junior people need to publish every piece they journal article they can, and usually are not in a position to pick and choose what (if anything) they put in edited collections. Senior people are in that position, and often don't face the publish-or-perish pressure that their junior colleagues do (though in a time of collapsing universities one can never play it too safe). But, to be frank, way fewer senior people read this blog--and, of those that do, I find it pretty unlikely that (if they currently value publishing in journals) they're willing to change their established publishing practice in order to make things ever so slightly better for others.

[grymes]

@crisis manager

Many universities conduct regular (in the case of my R1, yearly) reviews of faculty, and want you to publish something substantial--in some cases more than one thing--every year. I would love to spend a few years writing a truly excellent paper; my employment conditions simply do not allow it.

[Shen-yi Liao]

>> "Don’t feel you have to adopt every suggestion made by the referees."

Do the editors have data on the percentage of revisions submissions that (1) did not address some concern of a referee and (2) received a reject from that referee, and nevertheless were accepted by Analysis with the editors overruling the referee? How does this percentage compare to submissions that (1) did address concerns of a referee and (2) received an accept from that referee?

[Shen-yi Liao]

>> "Consider whether you really need to submit your paper to a journal. Would it be preferable to publish it in an edited collection, or just put it on your website? If the paper closely resembles your earlier publications, is there sufficient new material to justify publication?"

Assuming that the editors have been involved in the process of hiring candidates. All else being equal, how might they evaluate a candidate who has a paper published in Analysis versus a candidate who has just put that paper on their website?

[David Thorstad]

It's quite right that addressing every referee suggestion does not improve paper quality, but I worry that this trend is sometimes blamed on authors when what authors want to see is support from editors.

It's well known that referees hate being argued with and like seeing their comments addressed directly in print. Many authors also have the sense that referees who recommend rejection at leading journals will usually get their way. This leads, understandably, to authors avoiding argument with referees and addressing too many comments in print.

Could the editors provide authors with a credible guarantee of support for standing up to reviewers? The last time I stood up to a reviewer (for example, by suggesting that there isn't much cause to fret over the possibility that AI might soon murder us all) I lost a paper over it. I have to say I am not inclined to repeat the experience.

[jakub]

For me, Analysis epitomizes everything that is wrong with the current journal publishing system and with philosophy in general. These faults include: striving for shorter articles, competing for space, and an analytic style of thinking.
I'm forced by my employer to publish in journals like this. If there were no external pressure, I'd opt for books and edited collections. Most of my papers published in journals are worse than the original manuscripts that I submitted. One half of the paper is just to please the gatekeepers - that is, the editors and reviewers (proper argumentative style, connecting to current debates, citing trendy authors). And one half is missing because I was forced to remove all connections to other ideas. After getting a paper accepted, I'm tempted to add an acknowledgment: "Sorry for the occasional nonsense, but I needed to get this published."
Moreover, the idea, mentioned in these series, that people compete for space in a journal is preposterous. Journal publications are like prizes. Why so? We know from recent controversies that it is not because publishers restrict space. Quality and originality should be the goal, not competition.

[commenter]

so the suggestion is for reviewers to help fix the problem? Honestly this seems a little ridiculous. The problem is that there are too many good papers and too much snobbery about publishing. The obvious solution is for journals to publish more of these papers. I honestly don't understand this attitude of "we have to guard against the terrible papers that are coming!" as a reviewer myself I rarely find papers that should be outright rejected and feel pressured by journals to find some reason to reject the paper. Makes me less inclined to review until the journal editors do more (advocate for more papers in their journal, do more actually referring themselves, etc).

[State of the field]

Commenter
I think the field is really divided on this one issue (and many more of course) - unlike you I am not seeing many great papers when I referee. I have refereed 200 manuscripts for journals - many for the leading journals in the philosophy of science. I am not recommending rejecting papers because I have a concern for lack of space. I am recommending that unfinished, uninformed, and poorly argued papers not be published. But it seems that one person's journal half empty is another person's journal half full.

[Matt Weiner]

I was looking forward to seeing what the editors of a top journal had to say about how journals could help resolve the crisis in publishing--part of which includes philosophy journals' absurdly high rejection rates. But, even granting that you have no systematic solutions, this article seems to be entirely about what authors and reviewers can do to make life easier for journals. I hope that part 4 will make some suggestions what journals can do (and are doing) to help.

On the rejection rates, obviously I side with commenter; I often see papers I think are unworthy, I also often see papers that I think should not be published in their current form but that I would like to see developed. And I agree with Richard that the risk of not publishing good papers is higher than the risk of publishing bad papers; in philosophy it's not like people will accept faulty conclusions as true just because they're published (this follows from a more general principle about philosophers accepting other people's conclusions). I would say that going online does not completely solve the space/cost problem; Ergo is online-only, but our largest expense by far is still per-page costs for proofreading and typesetting.

I would also like to agree with grymes and Shen-yi that the advice to publish in edited collections and on your website is not helpful. Many philosophers would prefer to avoid the journal system, but do not get invited to edited collections and do not get professional credit or exposure for publishing on their websites.

[Michael Kates]

I wanted to echo Prof. Chappell's comment that it's bad practice to review a paper more than once. I understand the point the Editors of Analysis are making (especially about the problem of finding suitable reviewers in small subfields of philosophy), but I think that consideration is outweighed by the problem of gatekeeping. I don't have a monopoly on good judgment about the quality of a paper in my subfield, and neither does anyone else. Indeed, I think the inclination to review a paper more than once displays a certain form of intellectual arrogance. I've recently turned down two requests to review papers for precisely that reason, and I think it's good practice to question the reliability of one's own judgment on these matters.

[Assistant Professor]

Just here to amplify some great points raised in the comments:
- Accepting to review a paper you already reviewed (and rejected) elsewhere may be okay sometimes, but may also lead to one idiosyncratic cranky perspective tanking a perfectly good paper.
- Not submitting your work for peer review has real professional consequences so the advice about alternatives for people not-yet-established in the field is not useful.
- If editors don't want us responding to every piece of peer review feedback then editors please weigh in and help give us direction, and, you know, edit.

[constant reviewer]

I also agree with Richard and Michael--we should not be reviewing papers that we have already reviewed! The only exceptions I've made to this were when I initially had a fairly positive view of the paper, or where a quick skim of the paper suggests that I am likely to recommend acceptance or R&R for this version of it (I recently reviewed a paper that had been improved based on my comments at an earlier journal (where I had recommended an R&R but the editor had rejected the paper), and had no qualms about it because I recommended straight acceptance this time around--the paper was much improved. Other than those somewhat unusual cases, I think this is terrible practice. (And unlike many of the other commentators here, I have otherwise mostly agreed with what the editors have posted here, and as an aside just want to thank them for these illuminating posts!)

[Early Career]

To weigh in on the point about reviewing a paper you already reviewed. I personally sense a strong asymmetry on what I'd be willing to review a second time - one that tracks the point that it is worse not to publish good paper than to publish bad papers.

If I rejected it the first time, I would not accept the invitation, but if I'd recommended acceptance or revisions then I would be more willing to do so. I don't want to obstruct papers I think are bad, as I might be wrong. But if I found value in a paper then my turning down the invitation to review might unfairly work against that paper, especially if they get another reviewer who misses the value I saw.

[replies to refs]

Referees: Do you read the replies we write in response to your comments? Do you have a preference for the formatting of such reports?
Editors: how do you treat these in the revision process?

[Non-publisher]

Have to agree with jakub. Some people may just like seeing others competing crazily for a place in the journal they edit.