The Philosophers' Cocoon

Questions about suggesting reviewers

In our July "how can we help you?" thread, two readers asked questions about journals asking authors to suggest reviewers:

I work in philosophy of science, but recently I have started to submit papers in another subfield of philosophy. These are all journals with a good reputation. However, to my surprise, it seems that a pattern is being established: editors ask me to provide suggestions for possible reviewers. This never happened to me in philosophy of science. What's the best practice here? Naturally, I'd propose people I know (assuming they have not read the paper and they do not know I'm working on that specific topic), because I know that those people will take their job seriously. However, even if those people do not know that I'm writing that specific paper, they may recognize who I am. But this, I say, happens even in blind peer-reviewed, especially in highly specialized fields such as philosophy of science. Am I terribly wrong? - Posted by: Don't really know what to do

I have a question related to that of Don't really know what to do. Should a philosophy journal ask the author of a submission to suggest possible reviewers? What are the rationales for and again this? -Posted by: jack 

Another reader answered:

This is a common practice in science journals, to ask for (i) possible referees, and (ii) sometimes a list of people who you do not want to review your paper. In science it is common place to have only single blind - so the reviewer sees who the author or authors are. I publish regularly in an empirical field (call it scientific), and the key journal will not send your paper out for review unless you have your name on it.

However, while this may be a common practice in science, I wonder how readers feel about it, and whether anyone has tips for 'best practices.'

Are journal articles getting too long?

Brian Weatherson (Michigan) has a really interesting project featured at Daily Nous applying methods of 'topic modeling' to the history of philosophy. One of the many interesting findings Weatherson presents is that journal articles have gotten progressively longer in recent years, indeed longer than they have ever been. Here's the relevant graph:

In brief, while the average paper (50% percentile) was just less then 10 pages at points in the 1950's-60's, it is now double that. Similarly, while papers upwards of 30 journal pages were rare outliers throughout the 20th Century, now a full 10% of articles are upwards of 30 journal pages and outliers upwards of 35 pages.

Is this a problem? Are journal articles getting too long? Weatherson thinks so, writing, "This feels like a bad thing; articles are getting bloated, and we need to find a way to get them back to a reasonable length." For my part, I'm a bit torn for reasons outlined below. But I'm really interested to hear what you all think!

Neil Levy on peer-review

Neil Levy (Oxford, Macquarie) has some interesting reflections here.

Privately sharing an unused peer-review report?

In our most recent "how can we help you?" thread, Erik Magnussen writes: 

I was recently asked to review a paper and was halfway through doing so when I realized that the author had revealed their identity through a footnote. I quickly contacted the editor to let them know that anonymity had been breached, was thanked for bringing it to their attention, and was subsequently let go as a reviewer (as I think I should have been).

My question is this: would it ever be appropriate to forward my comments to the author in this type of case, or would this be an inappropriate interference in the editorial process? I thought the paper made a valuable contribution to the debate that it centered on, and I had a number of ideas about how the author's argument could be improved, which I had started to outline in my report. These may still be of use to the author, and I'm wondering if there is any definitive reason to keep them to myself.

To be clear, I have not forwarded anything to the author, and don't really have plans to. But it made me wonder about the ethics of doing so, and I know that the community here will have some thoughts on this. Thanks!

My initial reaction here was that this seems like a straightforward case: once a reviewer assignment is over, you're sort of free to do whatever you like (provided you otherwise stay within the bounds of general professionalism). However, after thinking about it for about 5 more seconds, it occurred to that the prudent thing here is probably to ask the Editor in Chief of the journal that commissioned the review to see if they have any objection. This occurred to me in part because, in my experience, some journals have an explicit policy that the peer-review process is supposed to remain confidential, and you never how exactly how this requirement may be understood. Sure, it may be unlikely that the author you reach out to you tattles on you for it (particularly if, as in this case, you let them know that you liked the paper)--but, I guess in general, my general maxim on professional uncertainty is this: "If you are in any doubt, ask before you act!".

But these are just my thoughts. What are yours?

Should COVID spur peer-review reform?

Long-time readers of the Cocoon may recall that I have advocated for reform to our discipline's peer-review system for about eight years now. In brief, our current system of peer-review seems to serve no one well. As I argued here and an informal survey I carried out seemed to support, our current system of anonymized peer-review:

1. Overwhelms editors with too many submissions.
2. Irritates and overwhelms reviewers with too many 'half-baked' submissions.
3. Frustrates authors with long and inconsistent turnaround times, peer-review reports of dubious quality, etc.

In addition to each of these problems, our current publishing system almost certainly impedes the timely dissemination of new research. COVID-19 is a prime example of the problem here. To see why, consider the following offhand remark that Jacob Stegenga makes in a guest post at Daily Nous today:

So far in the lockdown—May whatever-it-is-today—I am not aware of a systematic piece written by a philosopher of science on the pandemic or the policy response (if you know of one, I would be grateful if anyone reading this would direct me to it). Of course, I do not mean a journal article—our journals move too slowly for this—but I mean something deeper and more impactful than what normally appears in social media or blogs.

This is particularly striking to me. Over the past two months as the COVID-19 pandemic has progressed, articles in medical journals have been rapidly published (see e.g. here and here). Yes, there have been some COVID-related articles published in a few niche ethics journals (primarily interdisciplinary ones). But, as Stegenga's post illustrates, philosophy has by and large lagged behind other fields in disseminating new research on this world-changing topic. How many philosophy articles on COVID are stuck in anonymized review right now, as this pandemic rages around us? One can only imagine. Are there dangers to the rapid dissemination of research? Sure, particularly when it comes (as we all know by now) to epidemiological modeling. That being said, our current system of anonymized review--in addition to all of its other problems--is systematically poorly designed for this moment and world-changing moments like it. Philosophy should be on the front end of COVID, examining the myriad questions it raises in philosophy of science, ethics, etc. And yet, so far at least (as Stegenga notes), we seem to be decidedly lagging behind other fields.

Is it possible to go outside of our current publishing system to disseminate research more quickly--for example, by posting papers to the philarchive or publishing public philosophy in The Atlantic, etc.? Of course it's possible. But why should philosophers have to choose between these options when other fields don't? In physics, everyone expects unpublished drafts to be posted to the ArXiv prior to journal review. The same is true in medicine with the MedRxiv. In philosophy, on the other hand, there are no clear norms here: it's unclear from author guidelines at journals whether editors or reviewers frown on authors posting unpublished drafts to the philarchive. Indeed, for my part, I mostly avoid posting unpublished drafts there for this very reason. Will a journal editor or reviewer think it violates anonymized review? I have no idea, so for the most part I just avoid it. But why should this be how things are in our profession, when other professions have been using a more public alternative for decades?

The way I see, the time for peer-review reform in philosophy is now. How could such reform be achieved? Here's one possibility: the APA could take a clear stance in favor of authors posting unpublished drafts on the philarchive, recommending that journal editors and reviewers not hold it against others. And here's another possibility: journal editors could come out and state clearly that their journal endorses the practice. These changes, I believe, would benefit us all. A clear disciplinary norm that says it's just fine for authors to post unpublished drafts on the philarchive would incentivize authors to do so, enabling them to get more feedback on their papers, plausibly lessening the number of 'half baked' papers submitted to journals, reducing the number of revise-and-resubmits authors have to go through, and disseminating new philosophical arguments more quickly. We don't live in the 18th century anymore, and shouldn't keep using a peer-review system designed for that era. We live in a vastly different era, one where philosophers could make more timely contributions to public debate...if only we finally got down to business on peer-review reform.

Or so say I. What say you?

Reviewing book manuscripts

In our newest "how can we help you?" thread, a reader writes:

I'd be interested to know how people approach [book] manuscript reviewing. While publishers (in my experience) offer $100-200 to review a manuscript, the time commitment can be anywhere from ~5 hours to several days depending on the level of involvement of the reviewer. How long do people spend reviewing a manuscript (does this depend on the remuneration offered?) and what kinds of comments do they give (or what kinds of comments are expected and helpful)? I'd like to get a better picture of what is appropriate, as I have a good sense of how to review articles but not manuscripts.

An author responded:

You are not being paid to review a book manuscript. You are being given an honorarium, a small token of their appreciation. Keep in mind, if the press does not publish the manuscript, the whole process has cost them a lot. They have others to pay, etc.

The original commenter then responded:

Given that, what kinds of feedback do you think it's best to receive and give? I imagine line edits are too extreme, but that authors/publishers expect something more than a summation of the strengths and weaknesses of the manuscript. I could be wrong on this assumption though, so any help would be appreciated.

As someone who has both reviewed a number of book manuscripts and been the recipient of reviews, here's my own answer: I think the question of what kind of feedback it's best to receive and give depends primarily on the manuscript itself. Here's what I do:

1. I give the manuscript a quick read-through.
2. If it does not seem to me promising enough to warrant a contract, then I will quickly explain why (presses usually give reviewers forms with a series of questions to answer).
3. If it does seem to me promising enough to warrant a contract, then I will again quickly explain why, but then give the manuscript a close read, adding any feedback that I think would substantially improve the manuscript (including pointing out any serious mistakes).

When it comes to type-3 cases (where I think a contract is warranted), how much time I put in and the kind of feedback I provide depends on (A) how much time I have, and (B) how good of shape the manuscript is in. If the manuscript is in really good shape, then I may not feel the need to give detailed feedback (beyond any things that clearly stick out to me as things that need addressing). On the other hand, if manuscript is promising but really needs substantial work, I'll explain in fair detail why--much as I would in reviewing a journal article. 

As a quick autobiographical note, my reviews for journal articles are normally between 1-3 pages long, my reviews of book manuscripts normally similar in length, and the reviews I have received for book manuscripts I submitted generally similar in length as well. So, this seems to me about what to shoot for: something like the length and detail of a standard journal article review. Alas, I would say this can take anywhere from several hours to a few days, depending on the case. So unfortunately, my advice here doesn't really go beyond where the original commenter started.

Some Reasons I Reject Papers

One of the recurring discussions on this blog concerns paper rejections and the ongoing quest to get a paper past peer reviewers. The big question is how someone can increase their odds of getting a paper accepted to a good journal instead of just accumulating rejection verdicts. I thought it might be useful to share a little insight from my own experiences as a reviewer since I have given a lot of rejections and only a few revise-and-resubmit (R&R) verdicts, though all those papers got an “accept” from me after revisions).

When I first started peer reviewing a few years ago, I thought it would be very difficult to differentiate between papers that warranted R&R verdicts and those that should just be rejected. I wound up being wrong about this for the most part. (There has been only one paper I’ve reviewed that I think was genuinely on the borderline.) The reason I was wrong is that most papers I review feature glaring problems, some of which should be addressed well before the paper is submitted to a journal. Here’s a short list of the common reasons I have rejected papers as a peer reviewer:

• The paper ignores relevant literature. Sometimes, when a paper is in my wheelhouse, I can detect this immediately, but even in other cases, a quick search on PhilPapers often reveals that other philosophers have addressed the same issue as the author and that the author is either unaware of this or has ignored those scholarly contributions. I don’t expect authors to cite every existing source on their topic (which would be utterly impossible in certain subfields), but if robust and relevant debates on that topic are not even acknowledged in the manuscript, then the paper is seriously deficient in its level of scholarship.

Reader query on overdue reviews

A reader writes in:

I would like to ask about review practices. I have a submission under a decent journal. At one point, correspondences with the editors indicated that a review was overdue several weeks, they will contact the reviewer, and hope to come to a decision soon; later, at 7 weeks overdue, they mentioned that the reviewer needs more time. I’m not exactly sure how to phrase my question. Is this normal? And what should I do, if anything? What do reviewers and editorials normally do in similar situations? The context is that I’m going on the job market now, but I feel that I don’t have enough publications to be competitive.

I'm glad this reader wrote in, as this indeed appears to be a very common issue. I often see posts by friends on social media (including more senior members of the profession) relating similar stories. Generally speaking, the advice I've seen people given is to simply contact the editors again for an update after a reasonable amount of time has passed. However, if it has only been a few weeks after the initial exchange with the editors (as seems to be the case here), then it may make sense to wait a couple of more weeks (though I'm note sure). Another possibility, if things go on too long, is to simply withdraw the paper and submit it elsewhere, though I understand from experience why you wouldn't want to do that after you've already waited this long. I've been in a similar situation many times (including when I was on the job market in need of publications, and I never really felt like I knew what to do here aside from contacting the editors every month or so after the deadline. 

What does everyone else out there think? One thing to consider moving forward is to try to submit papers to journals with consistently good turnaround times. This 'Waiting for the Editor' webpage by Jonathan Weisberg is a great resource here, and I'm always surprised by how many people don't seem to be aware of it. 

Anonymizing when self-citing

In our most recent "how can we help you?" thread, anon postdoc writes:

I am working on a paper that relies in a couple of paragraph on an argument from a previous paper, already published. I was wondering how best to do this in a way that complies with blind-review. Some possibilities I am considering:

a. Cite myself as though it were another person.

b. Give a footnote and write '[deleted for blind review]' where the reference would go. (However, I have been told before, when doing this, that it is not sufficiently compliant with blind-review. Wish that my views were that famous!)

c. Appeal to some other argument in the literature that raises a similar concern, and give that prominence instead of my view. Then change this if the article gets accepted. (This could be used in combination of a or b).

This is a great query. I struggled with this very issue for a long time because I didn't know what the norms are. After seeing this discussed on social media, my sense is that the dominant norm is (a): citing yourself in the third-person, as though you are not the author. I am skeptical about whether this actually tends to work in terms of successfully anonymizing a piece--as my sense is that referees can probably still infer it's probably you a good proportion of the time (which I suspect may advantage prestigious, well-known authors and disadvantage lesser-known, less prestigious authors)--but still, my sense is that (a) is the norm.

Revise and resubmit (by David McCarthy)

The following is a revised version of an earlier guest-post by David McCarthy (University of Hong Kong).

This post discusses how to respond to an invitation to revise and resubmit from a leading philosophical journal. It is aimed at early career philosophers, and was commissioned by The Philosopher’s Cocoon. It is based on my own experiences as a reviewer and an author, but has been revised in response to commentary.

In light of feedback, I feel uncomfortably obliged to say that my own experience as a reviewer and an author is almost exclusively with very top generalist journals (e.g. Mind), leading generalist journals (e.g. Philosophical Studies), and very top specialist journals (e.g. Economics and Philosophy). These sorts of labels often just reflect perceived prestige, for whatever that’s worth. Nevertheless, very top journals, both specialist and generalist, do seem to treat revise and resubmit differently. Thus my suggestions may be less applicable to a broader range of journals, though I hope not.

I have not looked closely at data concerning how likely a revise and resubmit is to lead to publication at different journals. This is mainly because I think it is not especially important. A close reading of the revise and resubmit you are facing is much more useful, in my view, than looking at statistics. If you want to know what the weather is right now, you’re better off looking out the window than studying the morning forecast.

Journals being unable to find referees

In our newest "how can we help you?" thread, JR writes:

What to do when top journals reject papers without sending them for review not because the paper is bad but because they simply cannot find referees? My manuscript has been rejected from two top journals. One of the journals rejected the paper and the editor basically said that we do not have time to find competent referees for every paper and for that reason we have to reject some promising papers. Another journal rejected my paper after it has been "under review" for 6 weeks or so. They first contacted me and told that they have not yet found a referee for it because people keep rejecting referee invitations but that they will continue to find the referees. Just a day later I received another email saying that they have rejected my paper without any comments from the referees.

Is this practice common? Is it morally permissible? What an author can do to avoid this? Should I only submit papers to specialized journals that know more referees? (The problem is that if one does not publish in top philosophy journals, some philosophers think his/her work is not "real philosophy"). Should an author always recommend possible referees in a cover letter when submitting a paper? (at least some journals ask the author to recommend possible referees).

This is interesting, as similar phenomena happened to me for the first time recently as both an author and reviewer. In one case, an editor wrote to me apologizing for a long review time due to inability to find referees for a paper I submitted. Then, in another case, an editor for a different journal wrote to me asking me to review a paper because reviewers kept turning their requests down. In all my years, these are the first times I've encountered this phenomenon, and both instances happened in the past few months. I wonder what's going on, and whether this some kind of trend due to the professional conditions that David Velleman recently termed 'the publication emergency'.

Anyone else have any experience with this? On the one hand, it seems to me that members of the profession have a presumptive duty to take on review requests (viz. contributing their fair share to a system that plays a central role in our profession), so if people are turning down review requests left and right, then that's concerning. On the other hand, reviewing is 'unpaid labor' of sorts, so I know there's not much incentive for people to accept requests. But in any case, rejecting papers due to inability to find suitable referees seems super unfair to authors. So, I'm really not sure what to say on this. What do you all think? It might be good to hear from editors, reviewers, and authors on their experiences here!

Compromising anonymized review

I came across this illuminating post by a former journal editor yesterday, and was reminded of a disturbing discussion I happened across on social media a couple of weeks ago. In that discussion, a philosopher on social media began by sharing an interesting proposal: that journals should ask submitting authors for a list of people they have circulated their paper to, in order to help ensure that the journal selects reviewers who do not know the author's identity (viz. ensuring anonymized review). Although a number of people expressed support for the proposal (myself included), there was some pushback, including one bit that I found concerning. At one point, one commenter mentioned that they thought the proposal could prevent journals from finding the most qualified reviewers for a given paper. Their suggestion was that in very small subfields the people most qualified to review a paper might be ones who have seen the paper before and know who its author is. That person then added that they routinely accept reviewer assignments in cases like this, estimating that approximately 50% (!) of the time they review papers, they either know or strongly suspect the author's identity.

This isn't the first time I've come across claims like this on social media. Over the past few years, I have come across multiple comment threads on social media where reviewers and even some editors have said similar things. For example, in previous threads, I've some suggest that if a reviewer knows who a paper's author is, then the editor should decide whether the reviewer should accept the assignment. I have to confess that I find this very concerning. Anonymized review exists for a reason. The entire point of it is to prevent explicit or implicit biases toward or against authors from affecting editorial outcomes--so that work is judged on its merits. Reviewers or editors deciding to let a review occur when the identity of an author is known compromises this protection against bias.

On rejections for journal submissions

This is a guest post by Jon Robson, University of Nottingham for our series How to Write Philosophy.

Rejection is a big part of academic life. By far the most likely response to any job application, funding bid or journal submission is rejection. Here, I offer some thoughts on a very narrow aspect of this phenomenon: journal rejections (a very common phenomenon given that many of the most prestigious philosophy journals have an acceptance rate that’s well below 10%). Specifically, I will focus on the broadly professional aspects of responding to such rejections. Rejections from journals – especially those accompanied by ‘uncharitable’ comments – can also have negative effects on, for example, your psychological wellbeing. Advice for how best to deal with this is far beyond my limited area of competence. What I am competent (or at least experienced) in is thinking about the future of a recently rejected paper.

First, it’s important to remind yourself that a rejection merely represents the view of an exceptionally small number of your colleagues (often only a single colleague). To run the risk of comical understatement, philosophers don’t always agree. That one editor or a few referees judge your paper to be unworthy of publication is very poor evidence that this view will be the consensus amongst scholars in your area (indeed, I’d very be surprised if we could identify a single paper which every philosopher thinks ought to have been published). Yet, it’s similarly important to remember that the referee is a professional colleague and – at least when things go well – one with a reasonable level of relevant expertise. Often, they will have put a lot of work into their comments, ensuring that – again, all being well – the rejection is accompanied by useful feedback. It’s easy to be critical of referees (fun too), but many of them do an awful lot of diligent work for no real recognition.

So, what’s the best way to respond to rejections? Let’s deal with the easy cases first.

 

How to incentivize better author submissions?

In my past two entries in this series--as well as in other past discussions of peer-review--I have tended to focus on how editorial practices and reviewer behavior might be improved: to better ensure quicker and more reliable turnaround times, as well as better reviewer comments. However, it increasingly appears to me that this is to focus on only one part of the problem.

As David Velleman noted here--and others have noted as well (see below)--another central reason why peer-review has the problems is does is the behavior of authors. Journals are overrun with far too many submissions, many of which seem--to editors and reviewers--as though they should have never been submitted in the first place. This bogs down the entire system, making reviewers not only more difficult to find but also increasing their workload (as they have more papers to review). This in turn may be one significant cause of 'poor reviewer behavior', both in terms of long turnaround times and the quality of their reports. Indeed, there seems to be significant anecdotal evidence for this.

First, respondents to our poll on peer-review generally agreed with the idea that too many half-baked papers are sent out for review:

How to incentivize better peer-review reports?

A while back, I ran and reported the results of an informal survey on what readers think works well and not so well in peer-review. Although the results should be taken with a grain of salt, a number of trends emerged. One was that a vast majority (93.67%) of respondents agreed that 'journal turnaround times are too long and inconsistent.' In a follow-up post, I presented a possible proposal for how to address that issue: a central editorial system (perhaps at Philpapers) for journal editors to use that would give each reviewer a reviewer score, based at least in part on how quickly the reviewer responds to review requests and gets their reviews in. Whenever an editor gets a new paper, the system would then present the editor with the best reviewer matches--the final wrinkle being that the system would be designed to match papers with reviewers whose overall reviewer score is similar to the author's reviewer score. In other words, the system would match authors who are slow reviewers with other slow-reviewers, and authors who are fast reviewers with other fast reviewers.

Although this is obviously a fanciful system, at least at the present time (given that there is no such central system), it seems to me one that might be technologically feasible and work to everyone's advantage: authors, reviewers, and editors (particularly editors, as such a system would vastly simplify their task in finding reviewers!). By matching authors with 'reviewers like them', the system would plausibly incentivize many people to be better, faster reviewers--as there would be some incentive to be a better reviewer (namely, the incentive of getting matched with better reviewers as an author!). I also wonder whether this system might address a second set of issues that respondents in our poll cared about: quality of referee reports. As you can see below, many respondents in our poll thought that too many reviewer reports are incompetently, biased or needlessly aggressive:

Survey results on peer-review

Several days ago, I put up this informal survey to gauge readers' opinions about various features of peer review. I am going to leave the survey up so that people can still take it. But here are the results so far (n = 91), organized by what I will call the good, the ambivalent, and the bad.

As this is only an informal survey, the results should of course be taken with a grain of salt (and then some). Still, I found the results interesting and will be curious to hear readers' reactions.

The Good

64.1% of respondents agreed that anonymized review works well on balance, giving authors a fair chance (whereas only 23.07% disagreed).

An easier poll on peer-review

[Update: survey fixed!]

Following feedback on the poll on peer-review I rolled out the other day, I've tried to design an easier poll. HERE is a new survey that I think readers should have a much easier time with. Although it has 20 items, the items are so simple that I think it should take readers just a couple of minutes to click through. Hopefully we'll get some better data out of this one--and, although it's only an informal poll, my hope is that its results will give us some good material to discuss moving forward!

Reader poll: best & worst parts of peer-review

Based on readers' answers to my query on what works and what doesn't in peer-review, I've put together THIS POLL (poll suspended for revisions - see comments) for readers to rank features of the peer-review process in philosophy from best to worst. Although I'm by no means an expert at putting polls together, my hope is the results will give at least some rough indication of readers' attitudes. We'll see how it goes--and I plan to host discussion threads on specific issues moving forward--but I hope some of you take part!

Peer-review: what works and what doesn't?

Several months from now, I'm scheduled to take part in a discussion on our profession's peer-review practices: what works, what doesn't, and what might be improved. To prepare for the session, I'd like to run a new series here at the Cocoon soliciting readers' opinions on our peer-review practices and norms, and what (if anything) readers think might improve the process.

In today's post, I'd like to ask readers to weigh in on two initial questions:

1. What do you think currently works well in the peer-review process?
2. What do you think the biggest problems are with the peer-review process?

After hearing what readers think, my plan is to run a few informal polls, so as to gauge readers' overall satisfaction with the peer-review process, desire for change, and priorities if change is pursued. Then, after those polls, I hope to have a series of posts discussing some of specific issues in more detail, including more detailed debates about whether the peer-review process should change at all, if so how, and so on. 

So please do considering weighing in on questions (1) and (2). The more readers we hear from, the better discussion we are likely to have in the series overall!