Seems like the peer review system has given up the ghost

I am an associate editor for Ergo, and I was idly wondering last night what would happen once I reach Reviewer Z.

The situation is this: A minimum of two referee reports are needed for each paper that is sent out for review. Each referee is assigned a letter, and I've already burned through Referee invites A through F in a couple of days.

Each referee says the same thing "too busy". One wrote to me the following "Sorry, but this is the 13th refereeing request that I've received so far in 2022 [this was 15 February], and I've accepted most of the others. I try to limit myself to accepting about one per week on average."

Similarly, for the Journal of Analytic Theology, I along with the other executive editor and managing editor made a happy dance when finally, reviewer 8 for a very specialized paper said yes to a referee request. 

Just a couple of days ago, one of the journals I am on the editorial board for wrote to us

"Since the COVID-19 pandemic began it has been increasingly difficult for us to find referees to review papers that are submitted to our journal. At the same time, we receive more papers every year. It is common these days that we have to approach six or seven people before we are able to find someone to review a paper. In light of this situation, I would like to ask you for help and request you to review papers more frequently than before."

It has come to the point that the most suitable referee(s) for a paper are almost never available. It takes us so long to find reviewers, sometimes a month, six weeks or more (this situation is also exacerbated by the fact that many people don't respond to referee requests at all). This lengthens the span of time even more for the total review process. All sorts of small fixes don't work anymore, e.g., shortening the time that people get to review, asking for alternatives (this is still very much appreciated, but unfortunately alternative referees are just as unavailable). 

Of course, it could be that my editorial experience is atypical--I'm just one scholar.

But it is my strong suspicion that the peer review system is finally broken beyond reasonable repair. We've seen a slow worsening of the situation and the pandemic has finally broken the system. People are burnt out and overburdened, job candidates increasingly desperate. 

In light of this, we need to do something. We owe it to authors to change the system, I think, rather than tinkering at the edges. Any suggestions (ranging from proposals to overhaul the entire system and how to do this practically, such as by our own Marcus Arvan in this paper co-authored with Liam Bright and Remco Heesen) welcome. *

Comments

[the spanner in the spanner pile]

Jennifer Whitting's Slow Philosophy is the solution and despite being around for years now, no one has moved to implement it. If you can't increase the supply of referees, decrease the demand by decreasing the demand for publications.

https://dailynous.com/2015/12/31/a-modest-proposal-slow-philosophy-jennifer-whiting/

An extract:

"What would happen if the APA were to set guidelines for the maximum number of pages on which a tenure decision should be based? Suppose, for example, it were 100 pages – or even 175 – of what the candidate takes to be his or her best work. Of course no one would be prevented from writing – nor even from publishing – thousands of pages. But the letters of referees and departmental recommendations to deans, as well as the deliberations of departments, ad hoc committees and administrators, would all be restricted (officially at least) to the merits and demerits of the designated corpus.

Just think. Candidates would be encouraged to focus more on the quality (and less on the quantity) of their work. They might also have an easier time balancing work with other responsibilities (including civic and family ones). Referees, as well as colleagues expected to vote on the candidates’ cases, might have more time to read their work carefully and arrive at well-informed judgments – especially if as a result of the proposed limits these colleagues and referees had fewer book manuscripts and journal submissions on their hands."

[Jay]

Publishers should hire academics to be full time editors and journals should pay the referees as well. They won't do that as long as professors are willing to do editorial work for free. So if you are an editor of a journal, resign. And if you are not, do not become one until the publisher pays you. When all scholars do this, the publishers must pay to run the journals and the problem will be solved - or at least it gets better. If you spend 8 hours day finding referees you will find more than if you spend 1 hour per week. Plus the publication race would decrease since there would be more jobs to get (those payed editor positions).

Also, journals should use just one referee per paper. The referees just disagree with each other anyway.

[TheMan]

One worry with slow philosophy is that when you take your tenure case to the college or dean, they might not care (might even dislike) that the APA did a weird thing. Despite me really liking the spirit of the idea, this seems like a recipe for getting more out of whack with other humanities than we already are (tenure committees already have to explain why we tend to write fewer books… now why we refuse to submit so little?). I fear there is only so much of doing our own thing that administration is going to allow.

[Nicolas Delon]

Following up on slowing down the whole thing. Advice such as 'Have at least 5 papers under review at any given time' is surely not helping. It's not even great advice for job candidates since, like much advice of the sort, it faces a universalizing problem. If most job candidates follow the advice, which they seem to do, they are just barely keeping up, not taking the edge. It's a depressing coordination problem. No one wants to be the sucker who submits less while others keep submitting more. So yeah, an institutionally supported and/or cultural shifting of norms is badly needed.

FWIW I've never had more than three papers under review at a time (maybe four once), and that maximum was after I had started a tenure-track position. I'm very sorry for all the candidates who are under enormous pressure to keep submitting on and on.

[In a (temporary) non-academic position]

I was once told by a very successful (then early career) academic that there should be a literature review sentence or footnote around the 2nd page of a manuscript. This is to help editors find suitable reviewers.

I would love to hear from editors whether this helps. If it does indeed, I would like to recommend this practice to be more widely adopted, as this would be a mutually beneficial thing to do.

Absolutely don't think this would solve the original problem, but just saying with the hope to make the lives of editors easier.

[Bemused]

When Editors comment on how hard it is to find referees I suspect that part of the problem is the narrow pool from which they seek referees. Admittedly, my reasons for thinking this are merely anecdotal.

I am in the early to mid career transition and have multiple publications in top ten philosophy journals and many more in leading specialist journals. I have published work on several distinct topics and I am at an R1 university. However, I rarely get asked to referee papers (on average only about twice a year). I believe the reasons for this are that (1) I work at a relatively unknown university that is not in the US/UK/AUS/NZ and (2) I am not much of a social networker (i.e., I don't seek out influential philosophers at conferences or on social media and try to make myself known to them).

I know a few others in similar circumstances who also get very few referee requests. This leads me to suspect that many editors are too reliant on those in their own social network, or those who fit into their narrow conception of the "elite", when seeking referees to do the discipline's gatekeeping work. Certainly, something has gone wrong when an unpublished grad student at a Leiter top-10 department might find themselves getting more invitations to referee than me.

[suggestion floater]

Some suggestions:
1.Make it standard for authors to have the option to pass along referee reports from one journal to the next.

Of my last ~40 submissions, ~8 were R&Rs or conditional acceptances that editors rejected but not at the recommendation of the reviewers who recommended R&Rs or CAs. In the current system, I responded by sending each rejected paper to another journal where I'd bet it was assigned at least one new reviewer, and in most cases to another journal after it was rejected there... In the envisioned system, I'd have had and taken the standard option of sending the reports recommending R&R or CA for each subsequent submission. Relative to the envisioned system, I think the current one wastes a lot of reviewer labor. One thing that might help with implementation: encourage reviewers for top journals to indicate in their report if they think a paper would be suitable for publication in certain non-top journals.

(I seem to remember some publishers giving an option to transfer reports; but it's not a standard option.)

2. Don't reject R&Rs and CAs that do virtually everything requested by the reviewers and editors, especially not on the basis of new reasons.

This is a waste of author and reviewer labor. When I notice a journal doing this sort of thing, I allocate my reviewing labor to other journals.

3. Have journals publish stats for the ratio of publications per year to reviewer reports per year. Then encourage reviewers to think in terms of how many "review-ons" they're contributing, where the review-on value of a report for a journal is that journal's ratio (perhaps weighted by the reviewer's expected value for an additional publication in that journal), rather than number of reports they're writing.

If this somehow caught on, it'd incentivize reviewers to use their reviewing labor more efficiently (by reviewing for more-efficient journals, since they'd get more review-ons with less work that way); that'd in turn incentivize journals not to waste reviewer labor. It'd also give people a better sense of how much reviewer labor their consuming vs. contributing.

4.As a standard part of submission, ask authors to suggest lesser known referees with which they don't have personal connections. (I seem to remember the Journal of the APA doing something like this.)

5. Create a norm whereby you can opt out of refereeing guilt-free provided that you donate to a refereeing fund (perhaps managed by the APA). That fund would then just be used to hire philosophers to review papers as a full-time job. The norm might be something like: if you have tenure and submitted N times last year, it's okay to do have done fewer than 2N reports in the last year so long as you donate (2N - the number you reviewed) x 100 dollars.

[StickItTo]

A quick reply to TheMan's posts above. I agree entirely - I LOVE the concept of 'slow philosophy', and in an ideal world, I think that would be a fine way to run it. But I'm in an institution where administration regularly overrules both the department and external reviewers in P&T decisions (i.e. the only people with domain expertise). What they want to see is quantitative metrics - # of publications in top-X journals, H-indices and so forth. If someone went up with only 100 amazing pages, they'd likely deny the case.

[Wouldn't it be nice]

A few possible suggestions:

-In addition to Jay's suggestion, publishers could hire full-time in-house referees. These full-time referees might also be expected to publish a nominal amount, say once every few years or so to demonstrate that they maintain the expertise to referee papers. In addition to helping journals operate more quickly, these would be fantastic jobs. No teaching or service duties! Alternatively, they might be used similarly to post-docs for people circulating on the market.

-Publish more stuff. The journals are increasingly absurdly selective as they are crushed under the massive pile of submissions. But every rejection just means two more reports are going to be needed when the paper ends up somewhere else. Surely there's a lot of publishable stuff that doesn't need to be refereed so many times. Sometimes the process improves a paper, but often it doesn't. Editors and referees should just be less picky, and journals should do away with space constraints.

-In line with Bemused, editors should widen their referee pools. I'm a 5th year grad student at one of those PGR top 10 schools Bemused talks about. I've submitted to a fair number of journals and published a couple things. But I've never been asked to referee. I would be willing to do it, and so would many other grad students, I bet. So the narrowness Bemused talks about might not just exclude those at less Leiterific institutions. I wouldn't in general like early grad students to be referees, but most advanced grad students are publishing these days and would be qualified to referee. From my experience and other grad students I know, I doubt this potential pool of referees is being tapped sufficiently.

[Daniel Weltman]

Bemused: my view is even more based on anecdotes, and I'm in much less of a position to expect to be asked than you (I'm at a small non-R1 university and I haven't published as much) but I have the same impression. I see some extremely well-networked people who are very early in their careers and who talk about getting asked to referee stuff constantly. These people are often graduate students. Meanwhile I'm in my fourth year of being a professor at a non US/UK/AUS/NZ place and I am not a social networker at all, and I've been asked to referee 7 times total (9 if you count being asked to referee an R&R I previously refereed for).

I'm not complaining - I don't expect editors to have psychic powers, so asking people they know about because those people are good at networking seems quite reasonable to me. But something's got to explain why some graduate students at prestigious universities who have thousands of followers on twitter and so on are being asked to referee a bunch, and presumably the explanation is that many editors are finding refs through the social grapevine rather than through checking who has published on the topic lately on PhilPapers or whatever.

I think you and I might differ about how much this is part of the problem, although I don't really know. It's hard for me to judge what it's like to be an editor looking for referees since I've never been one. My sense is that asking more widely might just widen the number of people saying no. But, then again, I always say yes, so, maybe not. Maybe if requests were spread out more evenly rather than concentrated on good networkers, more people would be able to accept requests.

[Overseas Tenured]

I want to second Bemused. I do get a lot of refereeing requests at this point. But when I don't have time and recommend alternative referees, the editors almost always thank me for recommending people they didn't know. There are lots and lots of excellent young scholars in relatively obscure places, often outside the Anglophone world. For various reasons they also often do a better job at refereeing than the familiar go-to people (less overworked; read more than big shots who no longer keep up with the literature because reading other people's work is for losers; more humble; etc.). Editors should get out more and not send 95% of refereeing requests to the same 5% of scholars.

[Wake up]

I have to agree with Nicolas, some of the advice given at this site contributes to the problem. Somehow people think it is okay to just keep sending stacks of papers out to journals, and in many case they do not even bother changing them between submissions. I just finished reviewing a paper for a journal that I had reviewed a year and a half ago for another journal. I did not realize I had reviewed it before, until I finished (and found my old report, after I wrote the new report). The paper has the same fundamental errors in it. I review about 12-20 papers a year, and I am routinely asked to review twice as many. I also review book manuscripts, grant applications, and tenure files. People who want paid journal staff do not know what they are asking for. They have that in the sciences, and there are also publication fees.

[Also Bemused]

I strongly agree with Bemused. I'm in a similar situation (publications in top journals on a range of topics), and I also receive few referee requests. I also suspect this might be because I'm at a university outside of the UK/US/AUS/NZ circle and because of little social media presence (I'm very shy, what can I say?). So complaints to the effect that 'it's all broken' don't sound very plausible to me. (By the way, I strongly suspect I'm not on any editor's blacklist, in case anyone is thinking that.)

[Alex Grzankowski]

Good point, Bemused. I bet this is widespread. My knee jerk reaction was to open a google doc/sheet and start a list but quickly realised the needed specificity of specialty would be a pain. But then I remembered this:

https://philpeople.org/find-philosopher

Do editors use this? Do most people update their profiles in appropriate ways? This seems like a ready-made resource that could help tackle Bemused’s point. (And is useful for other sorts of things such as conference invites and syllabus building.)

Marcus, Helen - might a post prompting people to fill out their profiles for the sake of refereeing be useful along with a reminder to editors to use the resource if they aren’t already?

[Bill D'Alessandro]

Yeah, I find this imbalance pretty mystifying too. I'm a postdoc at a good place. I could referee pretty much any paper in philosophy of math, lots in philosophy of science, and plenty in metaphysics, epistemology and aesthetics. I've never turned down a referee request and never submitted a late report, and editors seem to find my reports helpful. But I get far fewer requests than the referee in Helen's story (although admittedly more than Bemused -- I got 12 last year). I could and would do more if asked.

[anonymous associate editor]

Many have suggested that editors widen their pool of candidate referees. There is already a mechanism for this: We ask referees who decline invitations to suggest alternative referees.

Overseas Tenured mentions this above, but I'd like to say a little more.

I’m on the board of a well-known generalist philosophy journal, where I handle about two new submissions per month. Unfortunately, about a third of my referee invitations are simply ignored, and another third are declined without offering alternative suggestions. When I do receive suggestions, about half of the time they aren’t helpful because they are completely obvious.

Here, then, are a few norms that it might actually be realistic to implement:

1. When you receive a request to referee, please respond.
2. If you decline, please suggest alternative referees, especially graduate students and other scholars whose expertise in the field might be harder to discern--say, by a philpapers search in relevant subject areas.

By the way, let me take this opportunity to thank those of you who are already doing this. It makes a big difference to both authors and editors, and we really appreciate it!

[EuroProf]

Just chiming in to second Bemused's point. I'm a full prof at a large university in continental Europe and I rarely receive requests for review from anglo journals. Maybe two or three per year. I mean, @Helen, we're even working in the same field, and I have never had a request from the Journal of Analytic Theology. Next time you can't find a reviewer, just send it to me, I'll do it.

[anon]

Relying on referees to make suggestions obviously isn't enough, given that this is something we already do, and that things are already going badly.

Is anyone trying to do something like advertise on Philos-L or some other popular place to get people to voluntarily indicate their willingness and competence to referee in certain areas? Or otherwise trying to proactively build a database rather than rely on social networks? You can always skim their CV if you don't know them.

I'm in a similar boat with Bemused and others. Several years out from the PhD, have published a couple of things per year, some top 10-20, some top 20-30, some specialist journals. And I get about two referee requests per year. Zero so far this year. I've received positive feedback from editors re: my reports, as well.

[Thomas Nadelhoffer]

Some people in this thread have noted that it is odd that they don't get invited to referee more that they do given their accomplishments, expertise, and interest. They're surely right that if editors cast a broader net, they would have an easier time finding referees. But this would only make the editor's job more difficult (since it's hard to know where to even start such a search). Why not encourage a practice whereby philosophers (of whatever stage of their career) send their CVs to journals where they would be willing to referee along with a brief description of which areas would be a good match given their interests? Editors could then compile a database of interested and qualified referees with whom they might otherwise be unfamiliar and who are openly willing to referee. In short, this inverts the usual approach. Rather than editors trying to find referees, you'd have potential referees expressing their interests and availability to editors.

[David Velleman]

Well, I'll repeat the suggestion that brought down a torrent of complaints on my head the first time around. Journals should refuse to publish the work of graduate students. The breaking of the dam that used to keep student work out of the submission pool -- i.e., the convention that graduate students should learn philosophy before attempting to do research -- has at least doubled the size of the pool. Even if the editors desk-reject the tarted-up seminar papers, they are left with plausible submissions that are nevertheless below what used to be the levels of originality and depth of the philosophical literature.
I know, I know, graduate students at non-Leiterific programs think that publication is their only path to employment. OK, then, let's consider a policy of refusing to publish submissions from graduated students at the top 20 programs. Problem solved.

[anon2]

Since it seems no one has mentioned this yet - add it to the list of reasons PhD programs need to slash admissions, yesterday!

[bop bop]

I want to echo some of the points that have already been made here. In particular, I do agree that editors are largely responsible for this.

For one, philosophy is a large enough discipline that this shouldn’t be a problem unless it’s the same pool of people being asked to review everything. Not that I’m complaining, but I’m an ECR with publications in good journals and I’ve never been asked to review a paper even though I am often invited to present at workshops and asked to comment on papers at invitation-only conferences. I suspect that as others have mentioned, part of the problem is that editors do not cast a wide enough net when trying to acquire reviewers. If you keep asking the same network of people to recommend other reviewers, you’re going to get the same answers.

Second, I have had similar experiences to ‘suggestion floater’ where the referees unanimously recommended R&R, conditional acceptance, or even acceptance and the editor rejected the paper anyway. I’ve had to send the same paper elsewhere, which wastes everyone’s time and takes up more labour from new referees who could have used that time to review other papers instead of a paper that was already reviewed carefully and favourably by someone else but rejected for some indiscernible reason by the editor. This practice sends the message that the editor does not value the expertise of the referees. Why would anyone waste their time reviewing a paper for these journals? Knowing what I know from the author’s side, I certainly wouldn’t.

The same thing happens when editors repeatedly enlist the same handful of lazy referees who produce needlessly nitpicky reviews that don’t say anything relevant to the content of the paper and recommend rejecting something they clearly haven’t bothered to read.

These practices all clog up the review system and all of them can be avoided with less laziness and more care from editors. I say this as someone who has also had absolutely fantastic experiences with other journals and their editors, so I know it’s possible and I know that the usual eDiTiNg iS tOo MuCh WoRk AnD eDiToRs ArE TiReD excuse is frankly kind of bull.

[Prof L]

I've refereed for a journal which publishes a list of their referees, right under the names of the editors. I think this is a really nice practice, and some minor incentive (like, really minor, but still) to referee and also to do a good job.

[Don't Want To Rat Anyone Out]

Two thoughts:

1) It shouldn't be that hard to make a quick bot that draws from philpapers to suggest referees. The bot would take the proposal as input, scan the citations, and use those citations to find clusters in the literature, and then churn out a list of referees. If this was possible nearly a decade ago, https://newsroom.ucla.edu/stories/this-research-project-had-a-happy-250116, surely something similar could be done today. Were I not a coding dinosaur (could I write this in C, or would I need to use C++?!), I might take a stab at it.

2) Although there is obviously a problem finding referees, etc., I think it is worth noting that the system also is working well in some ways. My experience reading in a cognate field is that the typical paper is a _lot_ worse than the typical philosophy paper, in large part because of the loose quality of refereeing. More gets published there, but a way higher fraction of published work is simply a CV line for them and a waste of time for me. Likewise, I participated in a pretty fancy interdisciplinary humanities working group, and I was really taken by how superficial the comments were, and I was shocked at how much of the writing was first-drafty. The archival work and the like was fascinating, but the actual argument was so hand-wavy and jargon-showy. And these were in many ways the best of the best!

So, while we should keep working to improve our system, let's get rid of the bathwater but keep our baby!

(I'm posting anonymously because I don't want to complain publicly about this program.)

[The Expanding Circle (Of Referees)]

Junior academic here. I accept almost all referee requests and in fact have been publicly recognized for my work as a referee by a professional philosophical organization. Yet I receive almost no requests from numerous journals I routinely submit/publish in, nor am I ever called upon to assess articles for top generalist journals.

Funny story: Although I had previously published in one of the top specialist journals in my subfield, I was curiously never asked to review... until I switched Unis and shared a hallway with the head editor of said journal.

[Ben Jones]

Bemused is spot on. Given journal editors' troubles finding reviewers, I'm always a little perplexed that a journal I published in never follows up to ask for a review. Obviously, a lot of problems contribute to current issues with peer review. But one of those problems is editors' availability bias. There is a pool of folks early in their careers who, though not social butterflies or at the fanciest universities, could competently and efficiently review more submissions than they now do.

[newly appointed]

I had this experience when I submitted to Ergo.

Referee A accepted, then I watched my paper go to B, C, and then D. After D said not to the review request, nothing happened for two weeks. Then Ergo wrote me saying: "Well, A suggested reject, and we just went off of that." If I recall, this might line up with their explicit policy, but it was a little frustrating for them to just run with the one report. As it happens, on reflection I came to think the paper needed a bit more work, and so I think it was the right verdict. Still...

[Josh Shepherd]

To all those Bemused - hit me up friends, I'll ask you to review stuff.

Just to chime in - like Helen I'm an area editor at Ergo. I'm also an associate editor at Phil Explorations. My experience in the last year or so is that to get 2 referees, I typically ask between 10 and 20 people. About 20% never respond, which really stalls the clock. (At the same time, I've reviewed about a paper a week so far this year. So I do get frustrated.) I search far and wide, and have discovered a lot of interesting philosophers just trying to find referees.
I get it, people are busy. I don't think the system is broken - I find all the recent fix-it proposals underwhelming or very sub-optimal - but it is getting harder for everyone to publish, to find referees, etc. My suspicion is that this is due in part to a bottleneck at 'top-x' journals (in addition to a higher volume of papers being submitted and philosophers submitting). If we didn't overrate venue as much as we do, submissions and willingness to referee would be more distributed amongst the journals we have. (We could probably use more well-run journals, though.)

I also like any proposal whereby rarely-invited philosophers become easier to find by editors. The truth, of course, is that the philosophy community is big and all of us parochially think we know more of it than we do.

[Why?]

To echo something above, it really doesn't make sense when editors go to the trouble of finding a reviewer only to ignore their advice (at least, in most cases). I had a paper R&R'd at a top journal, I revised it, sent it back, the reviewer said it should be published....and then the editor(s) decided to send it to another reviewer who then rejected it.

That was a massive waste of my time and of the first reviewer's time (reviewing the paper twice only to be completely ignored).

[Happily bemused]

I want to second Bemused's point since my personal experience is similar. I do not have social media accounts and work in a non-english-speaking country, and I got very few referee requests---zero in the past year. In particular, I have never been asked by any journals that I published in. (Can't complain, just a data point.)

[grymes]

Just to add another voice to the chorus, lest (what I take to be) the most important thread of this conversation be lost amidst the noise:

First, Bemused (and others) are clearly correct about the root problem: the pool of submitters is way bigger than the pool of regularly invited referees, for no good reason. (Even discounting grad students, to appease David Velleman, the pool of submitters with PhDs is way bigger than the pool of regularly invited referees with PhDs, for not good reason.) I just served on a search committee, and the diversity in the amount of reviewing done by different candidates is staggering--moreover, there seems to be a very strong correlation between Leiter-ranking of PhD-granting department and amount of early-career reviewing completed.

Second, Thomas Nadelhoffer is clearly correct about the easy solution to this problem: journals should actively and publicly solicit the CVs of people who would be interested in refereeing.

Am I wrong that this is all quite obvious? What am I missing?

[Tom2]

It also doesn't make sense--and I've had this experience--for editors to go to the trouble of finding two reviewers, have both those reviewers render "accept" verdicts, and then reject the paper.

If you're not going to publish the paper no matter its quality, why waste everyone's time like this?

[anon postdoc]

For what its worth, I got my PhD from a low ranked Leiter department. However, I've had decent success publishing and I'm now doing a research postdoc in a fairly notable institution/department (though it's not a philosophy department). I have multiple philosophy publications in my little niche area and I'd like to think that I could be included as an "expert" in this little corner of the philosophical world. Hence, I'm always puzzled that I've never onec been asked to referee for a philosophy journal... particularly seeing as I've refereed for several very highly ranked science journals. I'd love to know why this is the case, particularly as I'm a fairly willing reviewer.

[Mark]

I do wonder whether the pool of referees is being plumbed. I'm a TT at an R1, working in a mainstream area. For reasons I do not understand, I have only been asked to referee one paper in my life, and this was in grad school. I do submit to many of the mainstream journals, too, so it is not like I'm off everyone's radar. I don't think my experience is common but it would be surprising if it's unique.

[Caligula's Goat]

I'm not bemused (the poster) but I am bemused by this conversation. A lot of the discussion thus far has focused on singling out the wrong sorts of issues, in my view. We've been focusing on what we might call "people problems" instead of other kinds of problems. What's a people-problem? These are things that locate the woes of modern journals with the psychological failings of individuals:

1. Editors are not asking outside of their social networks
2. Graduate students and others should be volunteering more to review things by sending their CVs
3. We care too much about journal prestige and so create a bottleneck at the "top" journals
4. We should shame folks who don't review as much

I think that this is the wrong way of thinking not only about the problem but also about the solution. Instead of "people problems" I think we really need to focus more attention on structural/institutional problems (i.e., locating the institutional incentive structures that make reviewing an unattractive way of spending one's time). For example:

Why are editors and reviewers working for free? Why is the default assumption that we give our labor for free to what are usually for-profit corporations?

If academic publishing was an all-volunteer organization then maybe we can start worrying about naming and shaming people who don't play along but, as it is, I don't blame anyone for seeing the system as exploitative and choosing not to play along (not all journals are run by for-profit corporations, of course, but this is true for the majority of them). It may be an act of charity to edit or review for Taylor & Francis but it's supererogatory so long they don't put a dollar value on that labor (even if you expect to get something out of it like good feelings or good relationships with people in power).

Solutions aimed at changing our personal motivations seem to miss the point that we're all operating against a set of incentive structures imposed on us by our universities (and really a set of standards developed and set by elite instititions which are then copied by those wishing to enter that club).

Why, for example, do people favor many publications at prestige journals over fewer publications at less prestigious journals? Why is slow philosophy dead
in the water? Because, as others have already noted, many publications in prestige journasl are the coins of the realm and individuals put themselves at a massive disadvantage if they don't seek them out.

When the service of reviewing isn't valued by our institutions (it counts for literally nothing in our tenure and promotion process)and when providing that service comes at a cost (our time is zero sum, time spent reviewing is time not spent reading, writing, or spending with your family), it's little wonder we don't have too many people itching to review (and those of you who think you want to do a lot of reviewing may think differently once you're getting asked more often).

I think that these sorts of structural problems are a much better place to target critique and solutions than attempts to appeal to our sense of charity (i.e., to enrich Springer, Elsevier, Taylor & Francis and so on with free labor) or to shame people for not doing more free labor.

Structural, not personal, changes are what we need.

[Mark]

Follow-up: I should have read the other comments before posting. It turns out my experience is very much not unique.

[Shepherd]

Caligula's Goat: any structural change proposals?

Maybe you were suggesting paying editors and referees - in some sub-disciplines editors get paid pretty well, though not at any philosophy journals I know. I actually like the thought of journals (especially the for-profits) hiring a staff of reviewers. Hard to work out the kinks, and this might be feasible only at fairly specialized journals. But (and I've said this before somewhere) I like the (unrealistic) thought of reviewing being a career, and journals developing reputations in part because of the excellence of their reviewing staff.

[Greg]

Two small thoughts:
1. If the pandemic is the straw that broke the camel's back, then I don't think we should think of the current peer-review system as "broken beyond repair". Just speaking for myself: doing referee reports has never been in (say) the top-5 most important things in my life. But during the pandemic, the top-5 most important things in my life are all taking substantially more time and energy than they did pre-pandemic (for a variety of reasons).

So because all my time and energy is needed to do an adequate job with priorities (let's say) #1-7, that's the reason why I "just don't have time for" priorities #8 and below -- and refereeing papers is usually lower than #8 for me.

Perhaps I am being overly optimistic, but I hope that one day I will be able to exit 'survival mode' and return to some of my lower-priority goals.

2. I have been an associate editor at a niche journal, and I am currently a book reviews editor at another niche journal (being a book reviews editor is obviously a little different, but you are still asking people to put work into something that (a) is about someone else's research, and (b) isn't really substantively rewarded, either in money or prestige). Like many of the other editors in this thread, I spend a LOT of time trying to find reviewers. It takes me so long that I often feel like I am an inadequate editor.

My main way of finding reviewers is google: my professional circle of acquaintances does not completely overlap with the areas of the journals I've worked at, and I don't go to tons of conferences. So echoing an idea earlier in the thread: if you want to be asked to review more things (by me, at least), do what you can to have a full and informative web presence. Many times, I've had to guess whether someone would be an appropriate reviewer, because I couldn't find much information about them online. And I strongly suspect I am missing very good reviewers, because my google searches are not finding them.

[The Expanding Circle (Of Referees)]

Caligula: I agree that there 'institutional/structural' problems worth discussing. But the reality is: many other disciplines are able to make things work despite these shared institutional problems. Biology (to take a discipline outside philosophy I've published in) has responsive, agile journals, somehow. And there are many times more publishing biologists than philosophers (and biologists publish more per capita than philosophers). I wonder how they deal with these 'people problems'? Genuine question!

[Helen De Cruz]

Europrof, thank you so much. I think that this really shows the effect of regional clustering (I sometimes ask Dutch philosophers to referee for JAT but that's in part bc I know Dutch academia). Next time we have a paper in your area under consideration, I'll send you a referee request!

[Overseas Tenured]

I found David Velleman’s suggestion to ban student submissions eyebrow-raising, to put it gently. I must make a remark on that because it seems to me that it has not been picked up on yet.

Journals publish a lot of stuff by well-established people that is really bad in a way work by 5th-year grad students is rarely bad: sloppy, poorly argued, out of touch with the literature and often the same shtick they have been saying for 20 years. We all know what kinds of papers I’m talking about. There’s a special kind of bad that’s peculiar to well-established figures whose work, style and favorite moves are widely recognized even if they submit anonymously.

I’ve read papers by grad students and they never read like that. In fact, the worst papers I’ve read from top journals are never by grad students. They are usually from famous people who have nothing new to say or are used to much laxer refereeing standards than those that apply to the hoi polloi. If we ban anyone from submitting, it should be those with tenure. Not that I think we should ban anyone.

[Another Bemused Postdoc]

Just to add yet another datapoint, I am also (i) a reasonably published postdoc in (ii) a non-Anglo department (although with an Anglo-Ph.D., (iii) with little social media presence/networkign, (iv) working in a mainstream subfield of the discipline, and I receive a stunningly low amount of review requests--just a couple a year. In fact, weirdly enough, review requests have been *decreased* during Corona, having trickled to almost zero. Amusingly, a large amount of recent requests have been to review my own work by triple-blind journals.

I also have a suspicion that journal editors construe people's competence too narrowly. It's as if (to give a hypothetical example) I have published on the philosophy of love, but then people don't consider me a possible reviewer for the philosophy of sex, the philosophy of emotions, or social philosophy more widely. I almost always only get reviewer requests in the philosophy of love (to stay in the hypothetical example), but obviously I know the surrounding literature and fields quite well, so there seems huge untapped potential. I wonder whether that's me.

[Bemused^17]

Another bemused postdoc here. Unlike some of the above, I received my PhD from a highly ranked program and am now a postdoc at another highly ranked institution; I have published ~five papers with more under review, and have presented often at conferences, although I am not active on social media. I would love to review more, but receive about a request a year.

For some journals, I am asked to list keywords - either as part of my profile, or for an individual paper submission. I always assumed that these were used to find reviewers. So a couple of questions: (1) do editors actually use this feature? (2) is there any way in which authors can make their keywords more informative? On the latter point, I sometimes worry that the keywords I use are slightly different from those other people use to describe their paper, leading to a potential mismatch.

[Shaun Gallagher]

I edit an interdisciplinary (although mostly philosophy) journal (published by Springer, so it’s “for profit”, although I don’t see any of that). I want to respond to the “Bemused” thread, and specifically the idea that editors (some editors) draw on a narrow range of referees. It’s not the case at the journal I edit. The journal has been going for about 20 years and using the publisher’s Editorial Manager system (and I think it is similar for Scholar’s Choice software used by other giant publishers) we have built up a database of referees that, I guess number well over 1000, and likely close to 2000. As an editor I am constantly adding names to the list, usually drawing from the paper’s reference list and/or a google scholar search for authors in the relevant area. It’s true that when considering a paper I may immediately think of well-known scholars who would be good reviewers, but often (not always) I hesitate to invite them to review because they usually decline because they are too busy.

But I want to confirm that there is a crisis, since it does often require asking 7 or 8 or 10 people to get 2 to agree to review. And it would certainly be helpful (for editors and for authors) if people would simply respond “No” in a timely way, if they can’t do it.

Currently, I’m also special editing a Frontiers Psychology/Interdisciplinary issue. They organize the review process differently there. They have a kind of interactive process, but it amounts to a very similar result. The crisis is also apparent in that system. The journal wants a very quick turnaround and if invited reviewers don’t respond in a few days or so, the system starts to invite other reviewers. They provide a long list of what the system takes to be appropriate reviewers, from which the editor can choose. But I also found that the system itself sends out invites if the editor decides to give invited reviewers a bit more time to respond. On one recent paper it was difficult to find reviewers and I think there were over 30 invitations sent out before we got 2 reviewers to agree! Something is not quite right about that. I’ve also been on the receiving end of this sort of process, and still am. I get lots of invites to review. Since Jan 1 I’ve received 23 invitations to review from either journals or book publishers. I accepted 3. Amazingly 13 of these invitations were from Frontiers journals – sometimes I would receive 2 invitations in one day from Frontiers. I think most of these were simply automated invites; so I automatically decline unless the paper is so directly relevant to my research that I can’t resist.

I don’t think that having the journal hire a team of reviewers would work (even if we could get Springer to spring for that); unless the journal itself is extremely narrow in scope, and even then I would think an editor in some cases would need a highly specific expert, not on the team, to make a judgment. If the journal has a wide scope, then it would need a small army of reviewers to try to cover the field. I don’t have a realistic solution. Is there an ideal system that eliminates all problems?

[The Expanding Circle (Of Referees)]

Shaun: why think a team of reviewers need be paid? Most editors are not paid (in philosophy at least). They want to promote the subfield and want the delicious public recognition/influence that comes with editing a journal. I'm similarly motivated. I'd be willing to review 6-12 articles for my favorite specialist journal if I was in some capacity publicly recognized for my work as a reviewer.

There are a lot of well-meaning junior folks out there who are excited about philosophy and take the job of referee seriously. If only we could bring them into the fold somehow...

[Ben Bradley]

I took a half hour to collect a bit of info from Ergo that I thought would be useful.

For the last 39 papers that (i) were submitted more than a week ago and (ii) were not desk-rejected or resubmitted R&Rs, we have had 66 accepted requests and 96 declined requests. That is fewer than three declined requests per paper, and about a 40% accept rate. (I excluded R&Rs because the rate of acceptance of those requests is much higher, probably 90%.) Six papers have had five or more declines. No paper has had more than six declines. (I do recall one paper last year having around 20 declines. When that happens it can certainly feel like things are broken. My sense is this is very rare though.)

To me, this does not seem like a crisis requiring anything at all to be done differently, let alone things as drastic as some people are suggesting. But of course other journals might be having different experiences. Maybe some people are accepting requests from Ergo because we are open-access. If that is the case, then an obvious solution for other journals presents itself.

[Bill Harrison]

Follow-up to Overseas Tenured and David Velleman:

Firstly, I fail to see any legitimate problem with David's recommendation that journals should refuse to publish the work of graduate students at top-20 Gourmet Report programs. Given the data collected by APDA, those graduate students will very likely land a job at a top R1 institution even if they have no publications. Plus, it's very likely that the work they do submit for publication after landing the TT job will be published in one of the top-tier generalist journals, given how probable it is that the "blind" reviewer employed by the journal will already know the work.

Secondly, for journals that receive enormous amounts of submissions (thinking mostly of top-tier generalist journals here), I would like to recommend that editors prohibit authors who have published in their journal from submitting for at least 5 to 10 years. I cannot tell you how annoying it is to hear of a "backlog" of submissions at a journal but seeing the same authors publishing in it year in, year out. Punitive practices, like charging for submission or closing submission portals, are clearly not working. Maybe, a moratorium on submissions for those authors who have already published work in the journal will cut back on their over-submitting to the same journal.

If our collective aim as a discipline is to diversify and to be more inclusive, then we need to adopt practices that provide the space for up-and-coming scholars of diverse backgrounds to publish in more prominent places.

PS: Some journals prevent authors from submitting within one year of their most recent submission to the journal. That practice is also punitive because it further marginalizes the author who hasn't had the opportunity to publish in the journal. My recommendation avoids being overly punitive towards marginalized authors by permitting them to resubmit until such time as they have been published in the journal. Once someone has published in a journal, they cannot do so for a good period of time to make room for other marginalized authors.

[The Expanding Circle (Of Referees)]

@Ben Bradley: Thanks for this data. Would be really interesting to hear from other journals. In general, would be useful to get a more complete picture of how things stand in the discipline as a whole with respect to this issue. If APA Committee on the Status and Future of the Profession is going to try to address the dearth of referees (as was suggested on the Daily Nous discussion page), seems important to first establish that there is indeed a real problem to be addressed. Perhaps a separate post soliciting info from other journals would be helpful? Just a thought.

[Overseas Tenured]

I strongly disagree with Bill Harrison on how to cut down the journal backlog. Limiting submissions for everyone to the same degree seems fine to me; limiting only *successful* submissions I consider unacceptable, though.

The notion that unsuccessful submitters are "marginalized" and should be given more advantages over successful submitters is seriously misguided. Journals aren't egalitarian and shouldn't be. They should be meritocratic and should select papers for publication exclusively on the basis of philosophical excellence.

[StickItTo]

@Ben Bradley - I can confirm that I'm more open to refereeing for Ergo than other journals (to be clear, I accept almost all my referee requests, but I am more willing to referee for Ergo ceteris paribus). This is mainly because it is such a well-run journal and I want to do my bit to see it flourish (and part of that is its open-access model, although this is only one of the many respects in which it is exemplary).

[Charles Pigden]

To Caligula's Goat; I am all for changing people's behaviour by changing the incentive structures that generate the behaviour, but incentives have to be based on people's fundamental preferences. And the problem with refereeing is that it is mostly a chore and sometimes a torment. For me, at least it is a highly dispreferred activity (not absolutely always, as when I get asked to referee something really good, but *usually*). It would therefore be very hard to incentivise me to do more of it than I actually do. (I am in fact a fairly conscientious reviewer but it is duty not desire that binds me to the wheel of suffering.) Neither increasing the money nor adding to the prestige would be very much use, at least not in any realistic scenario. I would SO much prefer to spend the time working on my own projects or reading the tip-top stuff in my areas of interest (of which there is an abundance). But then I have a secure job and a decent salary.

The people who *can* be incentivised to take up the slack are presumably the young and the desperate. Hence the suggestion that junior philosophers can be recruited as paid referees. But it seems to me that even if you could set up such a system (highly doubtful) there would not be many takers for such a soul-destroying job. If the idea is that they could referee during the working day but could work on their now stuff in the evenings my guess is that a day of wall-to-wall refereeing would have a deadening effect on their creative powers and that anyway it would soon come to seem like a living hell. I am not sure how many takers there would be and I don’t think many people would be able to stand it for more than a couple of years.

More promising is the idea of tapping the reserves of under-used talent that have professed themselves willing to do their duty (but no more than their duty) as referees. Several ‘bemused’ contributors to this thread have professed themselves to be willing and well-qualified to their bit but are puzzled as to why they have not been asked. One answer I suspect is this. Editors use a) citations (in the submitted articles) and b) citation counts to select their referees. It often takes time to build up citations which means that younger scholars are less likely to be asked. (I have only one paper on truthmaker theory, but because it is well-cited I have often been asked to referee papers the subject. I also have an article on the analytic/synthetic distinction which in my view is equally good, if not better. But it only has a few citations which probably explains why I have never been asked to referee a paper on this topic. ) Now what well-cited but burnt out (or lazy) referees as frequently asked to do is to suggest substitutes (often younger substitutes ) who can stand in for them when they have had enough. That’s fine and dandy if the well-cited old philosopher is a well-connected, networked person at a prestige metropolitan university with a bevy of graduate students, recent graduates and colleagues working in their areas who they know and can vouch for personally. But what if the well-cited referee is an older counterpart of the original Bemused and ‘works at a relatively unknown university’ and is ‘not much of a social networker’? That’s my situation and my problem is that in only ONE of my ares of expertise (conspiracy theories ) can I confidently recommend a set of colleagues as stand-ins for myself when I have reached what I think of as my quota. Perhaps I am unusual in this; perhaps most older referees have plenty of recommendations that they could conscientiously make. But if I am not , this 9and the teh editorial habit of fist contacting the well-cited) might explain why so many of the wiling are not being asked to have their willingness exploited.

[Charles Pigden]

Reposted minus typos with some additional thoughts
To Caligula's Goat; I am all for changing people's behaviour by changing the incentive structures that generate the behaviour, but incentives have to be based on people's fundamental preferences. And the problem with refereeing is that it is mostly a chore and sometimes a torment. For me, at least it is a highly dispreferred activity (not absolutely always, as when I get asked to referee something really good, but *usually*). It would therefore be very hard to incentivise me to do more of it than I actually do. (I am in fact a fairly conscientious reviewer but it is duty not desire that binds me to the wheel of suffering.) Neither increasing the money nor adding to the prestige would be very much use, at least not in any realistic scenario. I would SO much prefer to spend the time working on my own projects or reading the tip-top stuff in my areas of interest (of which there is an abundance). But then I have a secure job and a decent salary.

The people who *can* be incentivised to take up the slack are presumably the young and the desperate. Hence the suggestion that junior philosophers can be recruited as paid referees. But it seems to me that even if you could set up such a system (highly doubtful) there would not be many takers for such a soul-destroying job. If the idea is that they could referee during the working day but could work on their own stuff in the evenings my guess is that a day of wall-to-wall refereeing would have a deadening effect on their creative powers and that anyway it would soon come to seem like a living hell. I am not sure how many would be willing and I don’t think many people would be able to stand it for more than a couple of years.

More promising is the idea of tapping the reserves of under-used talent that have professed themselves willing to do their duty (but no more than their duty) as referees. Several ‘bemused’ contributors to this thread have professed themselves to be willing and well-qualified to do their bit but are puzzled as to why they have not been asked. One answer I suspect is this. Editors use a) citations (in the submitted articles) and b) citation counts to select their referees. It often takes time to build up citations which means that younger scholars are less likely to be asked. (I have only one paper on truthmaker theory, but because it is well-cited I have often been asked to referee papers the subject. I also have an article on the analytic/synthetic distinction which to my mind is equally good. But it only has a few citations which probably explains why I have never been asked to referee a paper on this topic. ) Nor is this citation-based policy silly. That you have published in some area is does not mean that your work is all that good or that *you* would be any good as a referee. At least citations suggest that you have the respect of your peers.

Now what well-cited but burnt out (or lazy) referees are frequently asked to do is to suggest substitutes (often younger substitutes ) who can stand in for them when they have had enough. That’s fine and dandy if the well-cited old philosopher is a well-connected, networked person at a prestige metropolitan university with a bevy of colleagues and past and present graduate students, working in their areas who they know and can vouch for personally. But what if the well-cited referee is an older counterpart of the original Bemused and ‘works at a relatively unknown university’ and is ‘not much of a social networker’? That’s my situation and my problem is that in only ONE of my areas of expertise (conspiracy theories ) can I confidently recommend a set of colleagues as stand-ins for myself when I have reached what I think of as my quota. Apart from that I can only recommend (in the words of ‘anonymous associate editor’) names that ‘aren’t helpful because they are completely obvious’.

Perhaps I am unusual in this; perhaps most older referees have plenty of recommendations that they could conscientiously make. But if I am not, this (and the editorial habit of restricting first contacts to the well-cited) might explain why so many of the willing are not being asked to have their willingness exploited.