The Philosophers' Cocoon

Refereeing papers that are critical of you?

In our newest "how can we help you?" thread, a reader writes:

I was recently asked to review a paper that engages with some of my work. Since the journal asking for the review made the manuscript available to me, I was able to look it over before agreeing. I quickly discovered that the paper is very critical of one of my papers, and that it also includes a number of serious mischaracterizations and misunderstandings of my arguments (which I found very irritating). On the one hand, I feel the urge to accept the review request so that I can correct these mischaracterizations. On the other, I know that I am not a neutral party in this matter, and so I am also somewhat inclined to decline the review on account of my obvious bias. What do other people do when they encounter this kind of situation? Is it better to decline on account of my bias, or to accept on the grounds that I am an authority when it comes to what I say in my own papers?

These are very good questions. From what I've heard elsewhere on social media, my impression is that editors ask targets of critiques to referee papers fairly often. Further, in one way, this sort of seems like a good practice--for the journal has an interest in publishing good papers, and if anyone is well-placed to prevent mischaracterizations, then it is presumably the author being critiqued. On the other hand, it does (as the OP notes) introduce a kind of conflict of interest into the review process. After all, if anyone has an interest in not being embarrassed or their arguments disproven, it is the author whose work is being critiqued. So, I wonder whether editors have a way of spotting this. When editors receive a review from an author whose work is being critiqued, do they subject it to especially careful scrutiny, reading the paper for themselves to determine whether the referee's review is fair? 

I'm really curious to hear what readers think of this practice, as well as from editors--and on whether the OP should accept the referee assignment. What do you all think? 

How to get asked to referee?

In our newest "how can we help you?" thread, S writes:

How can I get papers to referee? Where to start? I am a junior researcher with a few publications, but not from EU or USA, and without much of a network. Any advices?

Several readers submitted replies. Michelle wrote:

[I]f the journals you are interested in refereeing for use Manuscript central or similar then create a profile and add you keywords. When the editor looks for reviewers you'll show up in the search. You can also email the journal and ask to be added to the reviewer list I imagine.

And Ref wrote:

You are less likely to be asked to referee for journal that you have not published in. So it is crucial that you publish in good places, if you want to referee paper for journals. I have refereed over 170 papers, many for leading journals in my sub-field - I am not junior. But that is largely because I have published in these journals (I also have three books with a leading press). Further, once you start refereeing, it helps if you are effective at getting useful reports in to the editor in a timely fashion. No kidding, most of the time I get my report in within 5 days of agreeing to referee the paper. Further, I write my reports for the editors - I given them a professional assessment of the paper, and provide justifications for my evaluation.

These both seem like good suggestions, though for what it is worth, I was asked to referee for a number of good journals early in my career before publishing all that much. What do you all think?

Contacting EICs in peer-review delays?

In our most recent "how can we help you?" thread, a reader asks:

A question about journal etiquette. I have a paper that has been under review at a journal for nine months. For the past two months, I have not been able to get any response from the managing editor about the status of my paper. My question is: is it ok to write to one of the chief editors of the journal about this? Also, the journal is triple-blind, but as far as I understand that applies to the area editor who is handling the submission.

Good question! Another reader submitted the following reply:

I got a paper off the backlog at [redacted], where it had been stuck for nine months with no response from the ME, by emailing someone from the editorial board. I tried the chief editor first, but they had not responsive either. But the editorial board member was very apologetic and got me a decision in a week. Extreme times call for extreme measures, I think. My only caveat: be extra polite if the CE works in your area or it's a journal you need to publish in at some point (I purposely chose a board member as far removed from my area as possible).

I'm curious what other readers think. Do any of you have any helpful tips or experiences to share?

How to know if you're a good referee?

In our most recent "how can we help you?" thread, a reader writes:

I referee more and more for journals, and there is this question at the back of my head everytime I send a report: How do I know if I do a good job as a referee? Or: What are good signs that you do a good/bad job as a referee?

A bit of context: When I send my reviews to editors, I usually also send them a message saying "if you need more information, or if my report is unclear, please let me know". I never hear back from the editors. When authors get an R&R, they are not dismissive of my comments, but this is hardly good evidence (as an author, being nice is part of the game!). We've all had experiences with bad referees. How am I supposed to know I'm not one of them? Maybe getting new invitations to referee from the same editors is a good sign. But perhaps they don't even keep track of this.

Good questions! In my experience, one of the best indications of one's performance as referee are how one's reports compare to other referees' reports. First, some journals have the (very good) practice of sharing all referee reports when an editorial decision is rendered not only with the author, but with other reviewers. As a reviewer, I've often found this to be very eye opening. On some occasions, I've seen that my worries were shared by other referees--which gave me some independent evidence that I had raised the right concerns about the paper. On other occasions, my reports seemed better than the other reports, giving more detailed feedback. However, on still other (thankfully rare) occasions, other reports led me to wonder whether I did a good job with my report (as they either understood something in the paper better than I did, etc.). For these reasons, I think the practice of journals sharing referee reports with all of a paper's referees is a great practice, and I only wish more journals did it (as in my experience, very few do). Finally, of course, sometimes journals share other referees' comments (or the author's reply to them) when authors submit revised papers for revise-and-resubmit verdicts. I find this to be really helpful for the same reasons. Other than this, though, my sense is that all one really has to go by is one's own internal sense of whether one is doing a good job.

What do you all think? What indications do you think a referee can have that they are doing a good job? And, are there things that journals should do in order to give one a clearer picture of how good of a job one is doing?

How are R&Rs handled?

(Update: comments now open!)

In our most recent "how can we help you?" thread, a reader writes:

When I re-submit a revised manuscript to a journal after an R-and-R decision, will they go to the editor first, or will they be sent out directly to the reviewers? I ask because I think I should clear something I've done in the revisions with the editor (a slight compromise of anonymity) before the reviewers see it.

Good question. My sense is that all submissions (both initial submissions and R&Rs) go to editors first, and then the editor decides whether to send it out to review, and if so, to whom. Sometimes (e.g. for an R&R with 'minor revisions'), editors may not send out an R&R to reviewers at all, deciding for themselves that you have adequately addressed the reviewers' concerns. Other times, the editor may send out the R&R to some of the reviewers but not others (my sense is that this can happen if one referee initially submitted an incompetent or unfair review: an editor might not be interested in hearing that referee's opinion again). Finally, sometimes editors may send out the R&R to all of the previous reviewers, or even (sigh) a new reviewer, if one of the original reviewers is unavailable. I guess my only concern about the OP is their note that they are interested in "clearing something" with the editor that might involve a "slight compromise of anonymity." If the journal is triple-anonymized (so that editors do not know the author's identity), this seems to me to be problematic. Although I can understand situations where this might be tempting (such as a reviewer misinterpreting some of the author's previously published work cited in the paper!), authors should still respect anonymous review in submissions and resubmissions, I think. But perhaps the OP can chime in and provide further context in the comments section.

Anyway, these are my thoughts. What are yours? 

How often should one referee for journals?

In our most recent "how can we help you?" thread, a reader writes:

I'm at the point in my career when I'm just starting to receive fairly frequent requests to referee journal articles, and I'm finding myself really unsure of how many of these I should be doing. Would cocoon readers be willing to share their own approaches to this? How much refereeing do people do? And do people have good rules of thumb for deciding when to say yes and no to referee requests? (E.g., a certain number of reports per month, or a certain number of reports per reports one has received oneself, etc?) This is one of those mysterious parts of the profession where I just have no idea if I'm pulling my weight, doing way more than I should be doing, or roughly getting it right. Thank you, everyone!

Excellent questions! One reader submitted the following response:

At one point, I was refereeing 20 papers a year, and working as an editor for PLOSOne, handling about 6 papers a year (which meant getting referees and making a final decision on papers). Now I am a co-editor on a journal, and I no longer do PLOSOne. So I referee only about 12 papers a year, and only in a select set of journals (the best in my sub-field or very highly regarded general journals). I also review a few book proposals and manuscripts a years for various presses. Without knowing your circumstances, I think if you are doing 12 a year, then that is fine.

Reviewing a bad paper for a journal?

In our newest "how can we help you?" thread, a reader writes:

For the first time is my (relatively short) career, I have been asked to review a paper that is very bad.

We all know the 'Reviewer 2' meme, and have probably heard of the research about the damaging effects overly that negative reviews can have on people - and I don't want to be that reviewer!

So, I'd love some advice focused specifically on how to write negative journal reviews without being too harsh in the tone. How does one best communicate a negative verdict without going over the line?

This is an excellent question, and I'd love to hear readers' answers. Although sometimes it can be hard to avoid being 'reviewer 2' (particularly if, as in the OP's case, the paper one is reviewing seems really bad), here are a few tips that I have found to be useful:

1. Use neutral, non-inflammatory language: As this comparative study found, something like 30% of reviews of philosophy papers contain inflammatory rhetoric. To avoid doing this as a reviewer, I've found that it can be useful to write one's review and then read through it multiple times to see whether there are any unnecessarily hostile words or phrases that I might replace with more neutral ones. 

2. Focus on detailed, dispassionate commentary rather than 'meta-commentary': Is the paper's argument confused? Instead of just saying it is confused (a meta-commentary), along with perhaps a short toss-off comment on why ("The paper commits all kinds of fallacies"), focus instead on showing the error(s) in question. Say, for example, "On p. 1, the paper claims X. On p. 2, it claims to derive Y from X. Yet Y does not follow. Here's why...". My experience as an author is that when one receives these kinds of explanations, a reviewer report can at least feel fair-minded--whereas if report merely says "the paper is confused", it can come across as unfair and needlessly hostile.

3. Set the paper aside for a week or so: Although as a reviewer one should presumably avoid unnecessary delays in completing one's review, it's amazing in my experience what a bit of 'distance' from a review can do. When you read your own review 'afresh', as it were, things can just pop out at you (like, "Wow...yeah, I should really tone that down").

4. Try to say a few laudatory things: Although it may seem hard to do in cases of especially bad papers, my experience is that papers that pass the desk rejection stage usually have at least some virtues. So, if there are any, mention them, particularly toward the beginning of the review.

5. Remind yourself of reviewer fallibility: I assume you have received referee reports that you thought were unfair, right? Well, those reviewers were presumably someone like you: someone who thought they were reading a 'really bad' paper. For this reason, I think it's helpful to consciously remind oneself of just how fallible we can be as referees. Try reminding yourself of this study, for example, which found a high number of classic (and 15 Nobel prize-winning!) economics papers to be rejected from journals, often for the most spurious of reasons (such as being 'trivial'). Or remind yourself of this story by Jason Stanley. Personally, I've found that reminding myself of how fallible we can be can really help one tone down a review. 

6. Read this discussion of bad reviewers by Michael Huemer: Don't reject a paper simply because you disagree with it, or even ultimately find the argument unconvincing. Your task as a reviewer should not be to expect perfection--especially given that, as we all know from graduate school, even the 'most important' papers in the field are usually flawed (or at least dubious) in all kinds of ways. Ask yourself instead, 'Is the paper interesting enough and good enough to publish, or at least give an R&R to?'

7. Imagine receiving the review yourself: I've heard people say they dread hearing from journals, and I empathize. Like most people I've talked to, I've received my share of harsh, meanspirited reviews. So, before submitting your review, ask yourself what it would be like to receive your review as an author. If your review feels needlessly 'hard to read', try to adjust its tone.

Anyway, these are just a few thoughts. Sometimes they can be admittedly hard to follow through on, particularly when one receives a paper to review that just seems incompetent in some way or other. Chances are, many of us will be 'reviewer 2' at some point in our lives. Nobody's perfect. But it is worth trying to do one's best to avoid it. Anyway, what do you all think? Do you have any good tips for avoiding being reviewer 2? If you do, I'm really curious to hear them!

Resubmitting a paper to the same journal?

In our newest "how can we help you?" thread, a reader asks:

Hello! I have a situation that I would like some advice. I submitted a paper to a special issue called for by a journal. The paper was rejected by the editor. However, the two reviews seem to both recommend a "revise and resubmit". I understand that the editor made the decision due to the fact that there is time pressure for the special issue. My question is, once I have the revised version ready, can I send it to the same journal (but obviously not for the special issue)? If so, should I make a note explaining all these, hoping that the editors reach out to the same two reviewers? Or should I just submit it somewhere else? Thank you!

These are good questions, and I have to confess that I've been in a similar situation. Anyone have any insight? Is it okay to resubmit a paper to the same journal as a 'normal submission' after it has been rejected for a special issue? Also, this brings to mind a more general question that I have seen raised from time to time: will journals ever consider a newer version of a paper they previously rejected, if for example enough time has passed (say, a few years) and the paper has been 'completely rewritten', substantially changing and improving its main argument? I've heard this question raised a few times by a few different people, but can't recall what the answer is!

Types of referee reports?

In our newest "how can we help you?" thread, a reader writes:

Maybe we could have a thread on 'types of referee reports'. I am working on a particularly frustrating R&R right now, and could use the entertainment. It maybe could also be instructive (?) on what not to do when writing a referee report?

Here, I'll start:

“You have not cited my book. No one else will tell you to cite my book, but I wrote it, someone published it, it’s on a glancingly related topic, so this lack of engagement with my work is a severe deficiency of your paper. Reject.”

“You have cited my book, but apparently you disagree with me. This means that you have not read my book carefully enough. Read it again. Reject.”

“You have talked about how X is related to Y, but have you considered that perhaps Z is related to Y? I do not know if it is, but neither do you, apparently, since you do not write about it, so … reject.”

“You are here writing about X. But have you considered mildly related issues Y and Z? Apparently you are ignorant of them. I would not consider publishing any paper on X that does not address Y and Z. Revise and resubmit […] This paper is too disjointed. What you write about Y and Z is nuanced and insightful, I am not sure why you discuss Y and Z in a paper on X. Reject.”

“I am just not convinced by this. Reject.”

“This is a great paper, unfortunately it’s on a topic I’m not super interested in. Have you considered rewriting the paper on a different topic, that is more related to my specific areas of interest and expertise? Thanks in advance. Revise and resubmit.”

This could be a rather fun and potentially illuminating exercise to end the week with, as it might help us all to reflect a bit on how we and others around us approach the practice of peer-review. I'll probably share an example or two in the comments section below, but I think I'd just like to open things up for comments first!

Tips for reviewing book manuscripts?

In our most recent "how can we help you?" thread, Kian Mintz-Woo writes:

I have been asked for the first time to evaluate a manuscript. The editor mentioned a 'small fee' or the equivalent in books. I am interested in the topic of the manuscript, know and like the (similarly junior) author, and believe I would be well-placed to comment on it. However, I know it is a massive time commitment and I am sure the 'small fee' is nowhere close to the value of that time.

What is a normal timeline for reviewing a manuscript (this is not very long--the author expects <100k words)? It is worth noting that the editor has mentioned that they can be very flexible abut the reviewing time. Are there other things I should keep in mind? (Should I basically just say no to manuscript reviews until I'm a mid-career academic?) Is there a reasonable range to expect the 'small fee' to be? (I am sure it's not worth haggling or anything like that, but even if I wanted to I don't know enough to do it.)

Is it relevant that I like the press and might pursue them in the future, as I think it is with journals (although, because of subject area, not this particular editor)?

Do I basically review the book like I would articles, just on a larger scale--i.e., track the argument as it unfolds over chapters? Or should I also be thinking about things like audience and sales (!) I would also add some copy-editing in, because I like to do that as well.

I searched through Phil Cocoon for advice on this and--while you have lots of discussions about preparing proposals--you don't (appear!) to have anything about evaluating them. Thanks for any thoughts or advice!

These are good questions that I suspect many early-career people might benefit from hearing good answers to. Here are some quick thoughts...

Should referees ever intervene with editors?

In our newest "how can we help you?" thread, a reader writes:

I refereed a journal submission a few months ago. I thought it was ok, but could use some revisions. I just looked at the submission and saw the other reviewer's comments. Basically, the reviewer is complaining that one paper, which was cited, wasn't engaged with enough, and that another paper that wasn't cited, which is somewhat relevant, should have been cited. That's it. On the basis of that they deem this submission as not sufficiently engaged with the literature. Since this is a niche topic I write about, I can confidently say that this is false. The submission does a decent job of engaging with the literature. My question is this: should I write to the editor to voice my disagreement with the other reviewer?

This is an interesting question. Another reader submitted the following reply:

No ... do not write to the editor to complain about the other referee. The two reports are supposed to be independent assessments. I do not like the practice of sharing referee reports like this anyway. But let the editor do her job now.

I have to confess to being more uncertain. If a referee seems like they are being unreasonable, I can see why another referee might want to intervene. But I also see the point that peer-review reports are supposed to be independent assessments. What do you all think?

Tailoring referee standards to journals?

In our newest "how can we help you?" thread, a reader asks:

I have a question about writing referee reports and matching one's standards to the selectiveness of the journal.

I'm relatively new to writing referee reports and am finding it difficult to know whether and how to match my standards of evaluation to the selectiveness of the journal. To what extent do others try to do this, and could anyone recommend any specific strategies? I feel like I just find myself asking whether the paper adds something important to the scholarly discussion as a whole (and whether it does so in a clear, effective way, etc.), and I don't really have any specific strategies for assessing things differently based on the particular journal's standards.

Great question, and I'm curious what readers think. I've refereed for journals all over the map (top 5 journals, unranked journals, etc.), and I definitely use different standards. I always try to hold authors to standards of soundness and cogency, for obvious reasons. So, for me, the main issue in terms of tailoring my standards to journals is my sense of the piece's overall significance. For example, I tend to be a bit puzzled when I see a paper in a top-5 journal that makes what I take to be a very small move in a well-worn philosophical debate. This isn't to say that small moves aren't worth publishing (surely, they can be). It just strikes me that particularly selective journals should be in the business of publishing particularly noteworthy works that make major contributions to ongoing debates, and that smaller, less significant moves probably belong in other journals. Is it possible for a very small move to make a major contribution? Sure, Gettier's famous paper in Analysis is a perfect example. But papers like this strike me as far and away the exception, not the rule. 

Anyway, that's my main deciding factor for tailoring my refereeing standards to journals. But, having read such journals, my sense is that other referees may tailor their standards in different ways. For example, some particular journals seem to place a priority on formalizations (viz. formal logic and decision theory). Personally, I think formalizations tend to be overused in philosophy--but I recognize this view is idiosyncratic to me. Anyway, what do you all think, particularly (but not only) those of you who have reviewed for different kinds of journals? How do you tailor your standards to different journals? Or, alternatively, how do you think referees should tailor their standards?

When editors ignore your reviews?

In our newest "how can we help you?" thread, Frustrated writes:

I have a question about good practices for reviewers. In the past few months this happened to me 3 times. I was asked to review a paper for a top journal in my AOS. This paper is really really bad. It's bad in the following sense: the topic is important, the thesis is interesting, but it'so poorly written and argued that, if I did something like that myself, I'd get for sure a rejection. However, I recommend major revisions. After a few weeks I got the revisions back, and nothing much has changed. I can see what the other reviewer has suggested, and it's minor revision. I then suggest rejection and the paper is accepted. From this I understand that the paper was either from an established scholar, or from a student of an established scholar. I feel I'm wasting my time by doing my diligent work of reviewer, given that the editor has already decided. Therefore, this is my question: when I have the feeling that something similar is happening again, shall I address the editor directly and ask how I should treat the paper? Or shall I just decline to review the revisions?

I'm glad Frustrated wrote in about this, as I think it's a great issue to discuss. Another reader submitted the following response:

Frustrated: is it possible that your standards are too high? Maybe this series of events has some other kind of takeaway. Three different editors have overridden your negative reports ... perhaps your standards are either too stringent or just out of keeping with what the editors are looking to publish. I've had an editor override a referee on one of my papers. The referee accused me of all kinds of weird things (like I'm *clearly* just a graduate student, and some senior scholar has made some edits to my paper) and seemed really irritated. The editor contacted me with this review and asked if I would make some changes. The paper was then sent out to a third referee, since the editor judged the second referee to have an overly harsh view of the paper. I was grateful to the angry referee despite all the mean-spirited commentary, since I was able to improve the paper. I am not an established scholar, nor is anyone running around wielding influence on my behalf. Did the editor act inappropriately? I don't think so, not at all. These are recommendations you are giving, not dictates.

This is more or less the reaction that I had to Frustrated's comment. If I am reading Frustrated's comment correctly, in just the past few months they have had three (different?) editors (at three different top-ranked journals?) ignore their negative referee reports in favor of more positive reports by other referees. What should Frustrated infer from this? While it's hard to know for sure without looking at the papers themselves, four things seem to me worth noting here:

1. Referees and editors at top-ranked journals tend to have very high standards (viz. high rejection rates).
2. In each of the cases the OP describes, at least one reviewer thought quite highly of the 'really really bad' paper in question, judging the paper to warrant only minor revisions.
3. The associate editors who handle a paper may be specialists in the area.
4. When they receive conflicting reports, these editors may read the paper themselves to decide which report(s) to trust.

Given these four things and given the fact that same thing has happened to Frustrated repeatedly, it seems to me that Frustrated should seriously consider that the possibility that (A) they are writing overly negative referee reports, and (B) that the papers they judge to be obviously really really bad aren't. After all, if the papers really were that bad, one would figure that someone else--either the other referees or the editors who read the reports and then the paper--would see this as well. But that's not what's happening here. In three separate cases, it seems, the editors have read the referee reports and the paper and decided for themselves that the other referees' positive reports are the ones to trust. 

Peer-review: keep, change, or abolish?

Darrell Rowbottom (Lingnan University) has drawn my attention to a new paper of his forthcoming in British Journal for Philosophy of Science in which he defends pre-publication peer-review against a recent argument by Heesen and Bright for abolishing the practice altogether. Readers may recall that over the years, I've tried to stake out a middle ground in this debate: arguing for substantial changes to peer-review, but not its abolition. In brief, my own position has been that philosophy should probably transition toward the publication model practiced in math and physics, wherein it is standard practice to:

1. Upload draft papers to the ArXiv (a preprint server similar to the PhilArchive).
2. Share, discuss, and critique unpublished drafts openly (a kind of crowdsourced peer-review that weeds out bad papers and improves good papers before journal submission).
3. Submit papers for peer-review at journals only after steps 1 and 2.

Although this model to a large extent undermines anonymous review (as people then know who has posted which papers to the ArXiv), my own sense is that the system works well and might help resolve a number of problems with our current peer-review system (including but not limited to: too many half-baked papers being submitted to journals, journal editors and referees being overwhelmed by submissions, etc.). So, then, we really have three options for peer-review in philosophy:

• Option 1 - keep the status quo: stick with the anonymized peer-review system we have.
• Option 2 - abolition: abolish pre-publication peer-review at journals altogether.  
• Option 3 - reform: keep peer-review at journals, but adopt a practice of posting papers on online servers first so that they can be openly vetted (e.g. evaluated, criticized, etc.) before journal submission.

Which option do readers most support, and why - especially, let's say, after reading the papers by Rowbottom, Heesen and Bright? For some additional food for thought, readers may want to check out this story shared over at Daily Nous  by Nathan Salmón (UC Santa Barbara).

What Do You Aim for in Referee Reports? (guest post by Finnur Dellsén)

This is a guest post by Finnur Dellsén, Associate Professor at the University of Iceland

As many people have pointed out recently, the peer review system relies on us accepting several referee requests for each paper we publish. So refereeing is a substantial part of our job as researchers, even if we don't get paid for it specifically. But how do we make sure we write helpful, insightful, and constructive referee reports? How do we improve as referees? We get all sorts of feedback on our own work --- from colleagues, editors and referees --- which helps us improve as researchers. We get much less, if any, feedback on our referee reports.

So when I was named Referee of the Year at the British Journal for the Philosophy of Science for 2020, I thought it would be a good opportunity to get the conversation going on how to write a good referee report (see my Facebook post). Below is a list of things that I aim to do in my reports (but I encourage others to add to the list in comments):

1. I always summarize the paper first in my own words. This lets the authors and editors know I carefully read and understood the paper. It also forces me to focus on the main contribution of the paper as opposed to minor concerns (which is important because of point 3 below).

2. I try to find as many nice things to say about the paper as I possibly can, especially when I recommend rejection. I highlight ideas and arguments that are especially interesting or compelling. This helps the author know what not to change when revising their paper (e.g. for another journal).

3. I try to make the amount of text in the review that I spend on each idea in the paper reflect its importance for the main thread of the paper. Every paper has things in it that I don’t agree with or find slightly unconvincing, but it’s usually not worth getting bogged down with those issues.

4. When suggesting changes, I say explicitly which of those changes I consider to be most important, even dealbreakers, and which ones are mere recommendations for the author to consider. This makes it clear to the author which changes to focus on.

5. I try to indicate clearly, if not to the author then at least to the editors, how confident I am in my recommendations -- not just about whether to publish, but also about individual points or recommendations for changes. In many cases, I am not an expert on all of the things the paper discusses, and when I’m not I try to say so explicitly.

6. Finally, referees can gently point to relevant work by underrepresented groups and/or work that is underrecognized in the field (e.g. because of prestige bias). I admit I haven‘t done this as often as I would have liked in the past, but I think it can do a lot to increase diversity and make philosophy more inclusive.

Those are some points that come to mind. What am I forgetting? What do you aim for in your referee reports?

(Thanks to Helen De Cruz for giving me this opportunity to discuss refereeing practices, and to the BJPS for the award!)

Drop-In Discussion on Refereeing and Publishing Norms & XPhi

Joe Ulatowski (University of Waikato) has drawn my attention to this interesting looking drop-in Zoom session, to be held on September 17th, 2020 at 5:00PM EDT. A quick description:

At a recent XPhi Under Quarantine, participants discussed the possibility of drafting a public statement concerning norms for journals on publishing non-xphi articles that were about xphi, but that didn’t contain any studies.  So, e.g., someone wants to float a hypothesis about what philosophical expertise could consist in but has no data to back up the claim that philosophers actually work that way. Or a paper that sketches an idea for a study, but doesn’t actually do the study it sketches. Or if a non-x-phi paper involves an out-there kind of vignette, can one reasonably ask of an author to go run a study on it? 

Because participants had an interest in talking over these issues, we have decided to host a special edition of X-Phi Under Quarantine. We will host a drop-in discussion session to consider not just the questions outlined above but any such question concerning refereeing norms or publishing norms that affect work in xphi. We ask participants to be prepared to talk about the following questions:

• • What are the norms of refereeing papers in experimental philosophy?
  • Do these norms follow the strict norms of general philosophy journals? If they are different, how so?
  • Is it sufficient for a referee to recommend a paper be rejected because they just don’t believe the data?
  • How much should data & methods be reported on exactly the model of a social science journal, when it is being published in a philosophy journal, especially a general one? (I.e., not Mind & Language, where one can perhaps presume a certain comfort level with such things on the part of the audience)
  • Should general philosophy journals even be publishing methodologically ambitious x-phi? Do they have the refereeing and editorial bench to do it well? one thing we could look to do is make up some sort of database of x-phi-comfortable philosophers and scientists who are willing to serve as referees for such work.
  • Pre-registration? Making data publicly available? Other anti-QRP practices that should be made more widespread?
  • Publishing studies that are adequate empirically but don’t really have anything new to say philosophically? Should null results be published? Should replications be published?

This seems great, and it also drew my attention to the New XPhi Blog, which I wasn't aware of!

How do you decide when to review for journals?

In our new "how can we help you?" thread, a reader writes:

I have been post phd for a few years now. At first, I thought it good to get some referee work under my belt. Now, I am getting a request to review at least once a month if not more. Today, I got two requests 15 minutes apart. My question is, how do people approach the decision whether to accept or reject a request to review? What are the considerations people take into account (besides fit to review)?

Great questions, and I'm very curious to hear what people think! As mentioned in the new thread, to make things more manageable on my end, I've asked readers not to respond in the comments section there--and so there are several comments that I have not approved. However, although I did not post them, I will still try to quote them anyway in new posts when relevant. So, on that note, Bryce responded:

A colleague once gave me this advice about reviewing: review as many papers per year as you submit. So maybe 1-3? The more you review, the more you will be asked for review. Just say no.

This is a rule of thumb that I have commonly heard too, and it seems to me a good one. As professionals, we should all do our part to contribute to what are, effectively, 'public goods' in the discipline. If you submit a lot of papers for others to review, then you should review a lot of papers. If, on the other hand, you submit few papers, then feel free to review more rarely. What does seem clearly wrong to me is to submit lots of papers and rarely review when asked. That's just free-riding. But let me add a few further thoughts.

The importance of narrative in publishing?

In our August "how can we help you?" thread, a grad student writes:

A question on publishing. I get the impression that papers at top journals usually not only aim to provide strong arguments for some claim, but also to provide some narrative about the literature and the place of the paper in it. (For instance, a recent paper defends the claim that view A is supported by view B, and also tells a story according to which opponents of A have long objected to it by implicitly appealing to B, though it is debatable that they have done so.)

I was wondering (a) how important this kind of story telling is in getting a paper published, and (b) if it is important, what are some strategies for developing a skill for it.

My own sense, for what it is worth, is that it is crucial to tell a compelling but accurate story of how the paper fits into the literature and why it makes an important new contribution. In fact, I've heard this is true across fields, not just in philosophy. I've also reviewed a lot of papers, and my sense is that two errors are common: (1) Authors taking for granted that the problem they are working on is important when it's not necessarily clear to anyone but a hard-core 'insider' how this is the case, or alternatively (2) Telling a false, poorly researched, and/or exaggerated story. As a reviewer, I find both a bit frustrating. If, qua 1, I don't see the point of a paper, then it can be hard to recommend acceptance. However, I find instances of (2) even more frustrating. In fact, it seems to me something of a trend in the papers I've reviewed in recent years. For example, one of the more common story-telling maneuvers I've seen seems to be this: "Everyone who defends theory-type T thinks P. This paper is the first to argue that if T, then ~P." The aim here, of course, is to make it look like the paper is doing something spectacularly new and original. The problem, though, in my experience that it's nearly always the case that the author is ignoring wide swaths of literature of T-theorists who clearly aren't committed to P, as well as at least a few T theory papers that explicitly argue ~P. So, for this most part, I've found this a really poor story-telling strategy. A good story in academic research has to be an accurate one: one that doesn't oversell what it's doing by ignoring relevant literature.

This brings me to a couple of strategy suggestions. First, to tell a good story you have to read. Don't just read 3 Famous T-theorists who defend P and then treat them as representative of the entire T-theory literature. Instead, survey the T-literature and find out who is committed to P, who isn't, and who has explicitly denied P--and show in your introduction that your paper contributes something new that they haven't. Second, I suggest getting to the 'hook' first and to not get lost in endless exposition. Try to start the paper with a bang, stating in the very first sentence or two what the paper does. And while accurately situating your paper in the literature is important, do it concisely. Finally, I tend to think it's really important to spell out early and again in the conclusion of your paper the broader implications of your argument for future research. Does your argument merely make a small intervention with no broader implications (in which case, your contribution is a pretty small one), or does it have implications for how T-theorists should theorize or implications outside of T altogether? Provided again you tell an accurate story, that can make your article come across as important (and thus, important to publish!).

But these are just my thoughts. What are yours?

Questions about journal responsiveness

In our August "how can we help you?" thread, one reader writes:

Have you ever withdrawn a submission from a journal due to a delay or a lack of responsiveness? What was the final straw?

And another writes:

Question about contacting journals and ghosting authors. I had a paper accepted for publication back in April In a top journal and was told I would be contacted by the journal press shortly after. A couple weeks went by without contact, so I dropped a kindly-worded follow up email to the managing editor, who said they would look into the matter and that I should expect an email soon (I.e., two weeks). Well, a month and a half goes by and still nada. I drop another kindly-worded email to the journal managing editor and still have yet to receive a response.

I know the process is glacially slow and I am deeply appreciative of all the work journal teams do. As a youngish, untenured prof, however, the silence is a bit unnerving. Some communication to the effect of, “We are backlogged and Covid slowed us down even further; you will be contacted some time in the next decade”, would help.
1. Is journal silence a common occurrence?
2. Suggestions about how to handle journal silence post-acceptance professionally?

While I wouldn't worry about journal silence in reader #2's case (once a paper is accepted, it's accepted - so don't worry!), there are I think some interesting questions to discuss here--among them:

• When does it make sense to withdraw a paper from a journal?
• How should authors and the profession more generally deal with non-responsive journals?

I'm not sure that I have any great tips here. When I was early-career (i.e. on the job market and pretenure), I took a lot of care to submit to journals with good reputations and turnaround times, which you can find out about here and here. Back then, I didn't have time to wait 6-12 months for journal rejections, so I vaguely recall withdrawing a few papers years ago when the review time was inordinately long and the journal was unresponsive. I'd usually send an email or two asking for an update, and if I didn't get a response after some amount of time I'd just send another email withdrawing the paper. But again, generally speaking, I just tried to stay away from those kinds of journals to begin with. What do you all think?

Questions about journal refereeing

In our August "how can we help you?" thread, a reader writes:

I'm wondering how to deal with the following as a journal referee:

1) The manuscript, which discusses a recent paper, just flat-out misses an argument that that paper makes for a particular premise P. It is clear that they missed it because in the manuscript, the author writes, "It isn't clear what the argument for P is." In fact the author of paper being discussed does argue for premise P. I know not only because I am the author, but *more importantly* because I pulled up the paper to make sure I argued for it. I want to flag this to the referee without making it obvious that I am the author being discussed. How can I do this?

2) The manuscript deserves an R&R. But its tone is perhaps overly polemical, even hostile at times. I would say this even if the paper that the manuscript discusses were not myself. For instance, the manuscript contains asides like, "...the paper, characteristically, does not contain real-world examples of [the philosophical phenomenon]". As a referee, how can I flag that these hostile-sounding asides are unnecessary? Again, without revealing that I am the author of the paper being discussed.

Another reader gave what I take to be good answers to both questions:

1) just cite the relevant passage and refer to yourself in the third person. They might guess it’s you, but they won’t have enough evidence for that to be a great guess (even though it’s true), and your responsibility is to avoid revealing your identity, not to avoid your identity being guessed at all costs.

2) the quoted passage makes it clear that the author strongly disprefers your methodology, but it doesn’t come across as hostile to my ear. (As a reader, I find that these sorts of asides can be helpful, as they make it clear what kind of philosopher I’m reading.) Of course, if it *isn’t* characteristic of your writing (or of the literature? out of context I’m a bit unclear on the charge) that it does not contain real-world examples, then that’s an error you can point out.

Indeed, the passage the referee refers to doesn't seem to me problematically hostile. It could well be that the author the paper is criticizing (the reviewer!) doesn't provide any real-world examples, and this could well be a problem with the reviewer's work. That being said, I do think there are two things here that a reviewer could potentially take legitimate issue with. First, suppose it's the case that real-world examples are irrelevant to the issue at hand. In that case, I think it could be fine for a reviewer to point this out and say that the aside is unwarranted. Second, suppose it's the case that in their work, the reviewer has provided real-world examples and the aside in question misleads readers into thinking they haven't. In that case, once again I think it would be fine as a reviewer to point this out. Still, I think it's important to be judicious here. I don't think it's a reviewer's job to defend their own work in a review. Correcting egregious misrepresentations is fine, as is criticizing errors that affect whether the paper should be published. But suppose the aside in question here is merely an aside, having no real bearing on the paper's main argument, and that the reviewer being criticized really didn't provide any real-world examples in their work. In that case, I think the thing to do is probably to let it go.

In any case, this brings us to a final issue the follow-up commenter raised above:

Here’s a related question of my own. I get that folks think that journal article tone needs to be policed. Even if that’s true, is it the referee’s place to do the policing? Shouldn’t the reviewer focus on scholarly merits? The editor, who represents the journal and its style, can police tone if they like.

For my part, I'm inclined to think that unless a journal gives clear directions on how referees should evaluate papers (which some journals do via worksheet questionnaires), it should be up to the individual referee how they evaluate a given paper. This, I think, is why it is called peer-review. I mean, I personally think the primary focus should be on the quality of the scholarship and arguments. However, we all have different views about what a good paper should look like, and some of our peers can think that presentation and tone can be important. For example, during my time as a graduate student I was often upbraided by some of my mentors for writing with too aggressive of a tone. In retrospect, I'm very thankful for it. Generally speaking (though I do think there can be warranted exceptions), I'm inclined to think that we owe each other 'scholarly respect' (or whatever you want to call it). Fortunately, when I review papers for journals, it nearly always seems to me that failures in this regard converge with scholarly failures--that, for example, haughty dismissals of an author tend to misrepresent them in ways that very much warrant correction on purely scholarly grounds.

But these are just my thoughts and experiences. What are yours?

What bugs you as a referee?

One of the things I've learned over the years is that some ways of writing philosophy papers seem to go over better with referees than others. I've also heard that there are ways of 'writing like a grad student' that referees can spot, and which tend not to work well. This certainly coheres with my personal experience, as I feel like I had to unlearn a lot of writing habits in learning how to publish successfully in philosophy. Unfortunately, I had to learn these things the hard way--mostly by trial and error. So, sometime soon (maybe early next week), I'd like to crowdsource some tips on both of these things from readers--as I think a list of tips might help our early-career readers. 

Before we get to that, I'd like to crowdsource from our readers who have served as journal referees on things that bug them when they review papers--as I think this might provide greater insight into the tips we crowdsource next week, as well as help us all better understand what kinds of things to avoid. For example, a while back, Trevor Hedberg provided this list of some reasons why he rejects papers. If you review papers for journals, what sorts of things routinely bug you, and why? Also, do the things that bug you lead you to reject papers, or are they just things that you wish authors would avoid (and which might, let's say, lessen the need for issuing revise-resubmit recommendations?).

Questions about suggesting reviewers

In our July "how can we help you?" thread, two readers asked questions about journals asking authors to suggest reviewers:

I work in philosophy of science, but recently I have started to submit papers in another subfield of philosophy. These are all journals with a good reputation. However, to my surprise, it seems that a pattern is being established: editors ask me to provide suggestions for possible reviewers. This never happened to me in philosophy of science. What's the best practice here? Naturally, I'd propose people I know (assuming they have not read the paper and they do not know I'm working on that specific topic), because I know that those people will take their job seriously. However, even if those people do not know that I'm writing that specific paper, they may recognize who I am. But this, I say, happens even in blind peer-reviewed, especially in highly specialized fields such as philosophy of science. Am I terribly wrong? - Posted by: Don't really know what to do

I have a question related to that of Don't really know what to do. Should a philosophy journal ask the author of a submission to suggest possible reviewers? What are the rationales for and again this? -Posted by: jack 

Another reader answered:

This is a common practice in science journals, to ask for (i) possible referees, and (ii) sometimes a list of people who you do not want to review your paper. In science it is common place to have only single blind - so the reviewer sees who the author or authors are. I publish regularly in an empirical field (call it scientific), and the key journal will not send your paper out for review unless you have your name on it.

However, while this may be a common practice in science, I wonder how readers feel about it, and whether anyone has tips for 'best practices.'

Are journal articles getting too long?

Brian Weatherson (Michigan) has a really interesting project featured at Daily Nous applying methods of 'topic modeling' to the history of philosophy. One of the many interesting findings Weatherson presents is that journal articles have gotten progressively longer in recent years, indeed longer than they have ever been. Here's the relevant graph:

In brief, while the average paper (50% percentile) was just less then 10 pages at points in the 1950's-60's, it is now double that. Similarly, while papers upwards of 30 journal pages were rare outliers throughout the 20th Century, now a full 10% of articles are upwards of 30 journal pages and outliers upwards of 35 pages.

Is this a problem? Are journal articles getting too long? Weatherson thinks so, writing, "This feels like a bad thing; articles are getting bloated, and we need to find a way to get them back to a reasonable length." For my part, I'm a bit torn for reasons outlined below. But I'm really interested to hear what you all think!

Neil Levy on peer-review

Neil Levy (Oxford, Macquarie) has some interesting reflections here.

Privately sharing an unused peer-review report?

In our most recent "how can we help you?" thread, Erik Magnussen writes: 

I was recently asked to review a paper and was halfway through doing so when I realized that the author had revealed their identity through a footnote. I quickly contacted the editor to let them know that anonymity had been breached, was thanked for bringing it to their attention, and was subsequently let go as a reviewer (as I think I should have been).

My question is this: would it ever be appropriate to forward my comments to the author in this type of case, or would this be an inappropriate interference in the editorial process? I thought the paper made a valuable contribution to the debate that it centered on, and I had a number of ideas about how the author's argument could be improved, which I had started to outline in my report. These may still be of use to the author, and I'm wondering if there is any definitive reason to keep them to myself.

To be clear, I have not forwarded anything to the author, and don't really have plans to. But it made me wonder about the ethics of doing so, and I know that the community here will have some thoughts on this. Thanks!

My initial reaction here was that this seems like a straightforward case: once a reviewer assignment is over, you're sort of free to do whatever you like (provided you otherwise stay within the bounds of general professionalism). However, after thinking about it for about 5 more seconds, it occurred to that the prudent thing here is probably to ask the Editor in Chief of the journal that commissioned the review to see if they have any objection. This occurred to me in part because, in my experience, some journals have an explicit policy that the peer-review process is supposed to remain confidential, and you never how exactly how this requirement may be understood. Sure, it may be unlikely that the author you reach out to you tattles on you for it (particularly if, as in this case, you let them know that you liked the paper)--but, I guess in general, my general maxim on professional uncertainty is this: "If you are in any doubt, ask before you act!".

But these are just my thoughts. What are yours?

Should COVID spur peer-review reform?

Long-time readers of the Cocoon may recall that I have advocated for reform to our discipline's peer-review system for about eight years now. In brief, our current system of peer-review seems to serve no one well. As I argued here and an informal survey I carried out seemed to support, our current system of anonymized peer-review:

1. Overwhelms editors with too many submissions.
2. Irritates and overwhelms reviewers with too many 'half-baked' submissions.
3. Frustrates authors with long and inconsistent turnaround times, peer-review reports of dubious quality, etc.

In addition to each of these problems, our current publishing system almost certainly impedes the timely dissemination of new research. COVID-19 is a prime example of the problem here. To see why, consider the following offhand remark that Jacob Stegenga makes in a guest post at Daily Nous today:

So far in the lockdown—May whatever-it-is-today—I am not aware of a systematic piece written by a philosopher of science on the pandemic or the policy response (if you know of one, I would be grateful if anyone reading this would direct me to it). Of course, I do not mean a journal article—our journals move too slowly for this—but I mean something deeper and more impactful than what normally appears in social media or blogs.

This is particularly striking to me. Over the past two months as the COVID-19 pandemic has progressed, articles in medical journals have been rapidly published (see e.g. here and here). Yes, there have been some COVID-related articles published in a few niche ethics journals (primarily interdisciplinary ones). But, as Stegenga's post illustrates, philosophy has by and large lagged behind other fields in disseminating new research on this world-changing topic. How many philosophy articles on COVID are stuck in anonymized review right now, as this pandemic rages around us? One can only imagine. Are there dangers to the rapid dissemination of research? Sure, particularly when it comes (as we all know by now) to epidemiological modeling. That being said, our current system of anonymized review--in addition to all of its other problems--is systematically poorly designed for this moment and world-changing moments like it. Philosophy should be on the front end of COVID, examining the myriad questions it raises in philosophy of science, ethics, etc. And yet, so far at least (as Stegenga notes), we seem to be decidedly lagging behind other fields.

Is it possible to go outside of our current publishing system to disseminate research more quickly--for example, by posting papers to the philarchive or publishing public philosophy in The Atlantic, etc.? Of course it's possible. But why should philosophers have to choose between these options when other fields don't? In physics, everyone expects unpublished drafts to be posted to the ArXiv prior to journal review. The same is true in medicine with the MedRxiv. In philosophy, on the other hand, there are no clear norms here: it's unclear from author guidelines at journals whether editors or reviewers frown on authors posting unpublished drafts to the philarchive. Indeed, for my part, I mostly avoid posting unpublished drafts there for this very reason. Will a journal editor or reviewer think it violates anonymized review? I have no idea, so for the most part I just avoid it. But why should this be how things are in our profession, when other professions have been using a more public alternative for decades?

The way I see, the time for peer-review reform in philosophy is now. How could such reform be achieved? Here's one possibility: the APA could take a clear stance in favor of authors posting unpublished drafts on the philarchive, recommending that journal editors and reviewers not hold it against others. And here's another possibility: journal editors could come out and state clearly that their journal endorses the practice. These changes, I believe, would benefit us all. A clear disciplinary norm that says it's just fine for authors to post unpublished drafts on the philarchive would incentivize authors to do so, enabling them to get more feedback on their papers, plausibly lessening the number of 'half baked' papers submitted to journals, reducing the number of revise-and-resubmits authors have to go through, and disseminating new philosophical arguments more quickly. We don't live in the 18th century anymore, and shouldn't keep using a peer-review system designed for that era. We live in a vastly different era, one where philosophers could make more timely contributions to public debate...if only we finally got down to business on peer-review reform.

Or so say I. What say you?

Reviewing book manuscripts

In our newest "how can we help you?" thread, a reader writes:

I'd be interested to know how people approach [book] manuscript reviewing. While publishers (in my experience) offer $100-200 to review a manuscript, the time commitment can be anywhere from ~5 hours to several days depending on the level of involvement of the reviewer. How long do people spend reviewing a manuscript (does this depend on the remuneration offered?) and what kinds of comments do they give (or what kinds of comments are expected and helpful)? I'd like to get a better picture of what is appropriate, as I have a good sense of how to review articles but not manuscripts.

An author responded:

You are not being paid to review a book manuscript. You are being given an honorarium, a small token of their appreciation. Keep in mind, if the press does not publish the manuscript, the whole process has cost them a lot. They have others to pay, etc.

The original commenter then responded:

Given that, what kinds of feedback do you think it's best to receive and give? I imagine line edits are too extreme, but that authors/publishers expect something more than a summation of the strengths and weaknesses of the manuscript. I could be wrong on this assumption though, so any help would be appreciated.

As someone who has both reviewed a number of book manuscripts and been the recipient of reviews, here's my own answer: I think the question of what kind of feedback it's best to receive and give depends primarily on the manuscript itself. Here's what I do:

1. I give the manuscript a quick read-through.
2. If it does not seem to me promising enough to warrant a contract, then I will quickly explain why (presses usually give reviewers forms with a series of questions to answer).
3. If it does seem to me promising enough to warrant a contract, then I will again quickly explain why, but then give the manuscript a close read, adding any feedback that I think would substantially improve the manuscript (including pointing out any serious mistakes).

When it comes to type-3 cases (where I think a contract is warranted), how much time I put in and the kind of feedback I provide depends on (A) how much time I have, and (B) how good of shape the manuscript is in. If the manuscript is in really good shape, then I may not feel the need to give detailed feedback (beyond any things that clearly stick out to me as things that need addressing). On the other hand, if manuscript is promising but really needs substantial work, I'll explain in fair detail why--much as I would in reviewing a journal article. 

As a quick autobiographical note, my reviews for journal articles are normally between 1-3 pages long, my reviews of book manuscripts normally similar in length, and the reviews I have received for book manuscripts I submitted generally similar in length as well. So, this seems to me about what to shoot for: something like the length and detail of a standard journal article review. Alas, I would say this can take anywhere from several hours to a few days, depending on the case. So unfortunately, my advice here doesn't really go beyond where the original commenter started.

Some Reasons I Reject Papers

One of the recurring discussions on this blog concerns paper rejections and the ongoing quest to get a paper past peer reviewers. The big question is how someone can increase their odds of getting a paper accepted to a good journal instead of just accumulating rejection verdicts. I thought it might be useful to share a little insight from my own experiences as a reviewer since I have given a lot of rejections and only a few revise-and-resubmit (R&R) verdicts, though all those papers got an “accept” from me after revisions).

When I first started peer reviewing a few years ago, I thought it would be very difficult to differentiate between papers that warranted R&R verdicts and those that should just be rejected. I wound up being wrong about this for the most part. (There has been only one paper I’ve reviewed that I think was genuinely on the borderline.) The reason I was wrong is that most papers I review feature glaring problems, some of which should be addressed well before the paper is submitted to a journal. Here’s a short list of the common reasons I have rejected papers as a peer reviewer:

• The paper ignores relevant literature. Sometimes, when a paper is in my wheelhouse, I can detect this immediately, but even in other cases, a quick search on PhilPapers often reveals that other philosophers have addressed the same issue as the author and that the author is either unaware of this or has ignored those scholarly contributions. I don’t expect authors to cite every existing source on their topic (which would be utterly impossible in certain subfields), but if robust and relevant debates on that topic are not even acknowledged in the manuscript, then the paper is seriously deficient in its level of scholarship.

Reader query on overdue reviews

A reader writes in:

I would like to ask about review practices. I have a submission under a decent journal. At one point, correspondences with the editors indicated that a review was overdue several weeks, they will contact the reviewer, and hope to come to a decision soon; later, at 7 weeks overdue, they mentioned that the reviewer needs more time. I’m not exactly sure how to phrase my question. Is this normal? And what should I do, if anything? What do reviewers and editorials normally do in similar situations? The context is that I’m going on the job market now, but I feel that I don’t have enough publications to be competitive.

I'm glad this reader wrote in, as this indeed appears to be a very common issue. I often see posts by friends on social media (including more senior members of the profession) relating similar stories. Generally speaking, the advice I've seen people given is to simply contact the editors again for an update after a reasonable amount of time has passed. However, if it has only been a few weeks after the initial exchange with the editors (as seems to be the case here), then it may make sense to wait a couple of more weeks (though I'm note sure). Another possibility, if things go on too long, is to simply withdraw the paper and submit it elsewhere, though I understand from experience why you wouldn't want to do that after you've already waited this long. I've been in a similar situation many times (including when I was on the job market in need of publications, and I never really felt like I knew what to do here aside from contacting the editors every month or so after the deadline. 

What does everyone else out there think? One thing to consider moving forward is to try to submit papers to journals with consistently good turnaround times. This 'Waiting for the Editor' webpage by Jonathan Weisberg is a great resource here, and I'm always surprised by how many people don't seem to be aware of it. 

Anonymizing when self-citing

In our most recent "how can we help you?" thread, anon postdoc writes:

I am working on a paper that relies in a couple of paragraph on an argument from a previous paper, already published. I was wondering how best to do this in a way that complies with blind-review. Some possibilities I am considering:

a. Cite myself as though it were another person.

b. Give a footnote and write '[deleted for blind review]' where the reference would go. (However, I have been told before, when doing this, that it is not sufficiently compliant with blind-review. Wish that my views were that famous!)

c. Appeal to some other argument in the literature that raises a similar concern, and give that prominence instead of my view. Then change this if the article gets accepted. (This could be used in combination of a or b).

This is a great query. I struggled with this very issue for a long time because I didn't know what the norms are. After seeing this discussed on social media, my sense is that the dominant norm is (a): citing yourself in the third-person, as though you are not the author. I am skeptical about whether this actually tends to work in terms of successfully anonymizing a piece--as my sense is that referees can probably still infer it's probably you a good proportion of the time (which I suspect may advantage prestigious, well-known authors and disadvantage lesser-known, less prestigious authors)--but still, my sense is that (a) is the norm.

Revise and resubmit (by David McCarthy)

The following is a revised version of an earlier guest-post by David McCarthy (University of Hong Kong).

This post discusses how to respond to an invitation to revise and resubmit from a leading philosophical journal. It is aimed at early career philosophers, and was commissioned by The Philosopher’s Cocoon. It is based on my own experiences as a reviewer and an author, but has been revised in response to commentary.

In light of feedback, I feel uncomfortably obliged to say that my own experience as a reviewer and an author is almost exclusively with very top generalist journals (e.g. Mind), leading generalist journals (e.g. Philosophical Studies), and very top specialist journals (e.g. Economics and Philosophy). These sorts of labels often just reflect perceived prestige, for whatever that’s worth. Nevertheless, very top journals, both specialist and generalist, do seem to treat revise and resubmit differently. Thus my suggestions may be less applicable to a broader range of journals, though I hope not.

I have not looked closely at data concerning how likely a revise and resubmit is to lead to publication at different journals. This is mainly because I think it is not especially important. A close reading of the revise and resubmit you are facing is much more useful, in my view, than looking at statistics. If you want to know what the weather is right now, you’re better off looking out the window than studying the morning forecast.

Journals being unable to find referees

In our newest "how can we help you?" thread, JR writes:

What to do when top journals reject papers without sending them for review not because the paper is bad but because they simply cannot find referees? My manuscript has been rejected from two top journals. One of the journals rejected the paper and the editor basically said that we do not have time to find competent referees for every paper and for that reason we have to reject some promising papers. Another journal rejected my paper after it has been "under review" for 6 weeks or so. They first contacted me and told that they have not yet found a referee for it because people keep rejecting referee invitations but that they will continue to find the referees. Just a day later I received another email saying that they have rejected my paper without any comments from the referees.

Is this practice common? Is it morally permissible? What an author can do to avoid this? Should I only submit papers to specialized journals that know more referees? (The problem is that if one does not publish in top philosophy journals, some philosophers think his/her work is not "real philosophy"). Should an author always recommend possible referees in a cover letter when submitting a paper? (at least some journals ask the author to recommend possible referees).

This is interesting, as similar phenomena happened to me for the first time recently as both an author and reviewer. In one case, an editor wrote to me apologizing for a long review time due to inability to find referees for a paper I submitted. Then, in another case, an editor for a different journal wrote to me asking me to review a paper because reviewers kept turning their requests down. In all my years, these are the first times I've encountered this phenomenon, and both instances happened in the past few months. I wonder what's going on, and whether this some kind of trend due to the professional conditions that David Velleman recently termed 'the publication emergency'.

Anyone else have any experience with this? On the one hand, it seems to me that members of the profession have a presumptive duty to take on review requests (viz. contributing their fair share to a system that plays a central role in our profession), so if people are turning down review requests left and right, then that's concerning. On the other hand, reviewing is 'unpaid labor' of sorts, so I know there's not much incentive for people to accept requests. But in any case, rejecting papers due to inability to find suitable referees seems super unfair to authors. So, I'm really not sure what to say on this. What do you all think? It might be good to hear from editors, reviewers, and authors on their experiences here!

Compromising anonymized review

I came across this illuminating post by a former journal editor yesterday, and was reminded of a disturbing discussion I happened across on social media a couple of weeks ago. In that discussion, a philosopher on social media began by sharing an interesting proposal: that journals should ask submitting authors for a list of people they have circulated their paper to, in order to help ensure that the journal selects reviewers who do not know the author's identity (viz. ensuring anonymized review). Although a number of people expressed support for the proposal (myself included), there was some pushback, including one bit that I found concerning. At one point, one commenter mentioned that they thought the proposal could prevent journals from finding the most qualified reviewers for a given paper. Their suggestion was that in very small subfields the people most qualified to review a paper might be ones who have seen the paper before and know who its author is. That person then added that they routinely accept reviewer assignments in cases like this, estimating that approximately 50% (!) of the time they review papers, they either know or strongly suspect the author's identity.

This isn't the first time I've come across claims like this on social media. Over the past few years, I have come across multiple comment threads on social media where reviewers and even some editors have said similar things. For example, in previous threads, I've some suggest that if a reviewer knows who a paper's author is, then the editor should decide whether the reviewer should accept the assignment. I have to confess that I find this very concerning. Anonymized review exists for a reason. The entire point of it is to prevent explicit or implicit biases toward or against authors from affecting editorial outcomes--so that work is judged on its merits. Reviewers or editors deciding to let a review occur when the identity of an author is known compromises this protection against bias.

On rejections for journal submissions

This is a guest post by Jon Robson, University of Nottingham for our series How to Write Philosophy.

Rejection is a big part of academic life. By far the most likely response to any job application, funding bid or journal submission is rejection. Here, I offer some thoughts on a very narrow aspect of this phenomenon: journal rejections (a very common phenomenon given that many of the most prestigious philosophy journals have an acceptance rate that’s well below 10%). Specifically, I will focus on the broadly professional aspects of responding to such rejections. Rejections from journals – especially those accompanied by ‘uncharitable’ comments – can also have negative effects on, for example, your psychological wellbeing. Advice for how best to deal with this is far beyond my limited area of competence. What I am competent (or at least experienced) in is thinking about the future of a recently rejected paper.

First, it’s important to remind yourself that a rejection merely represents the view of an exceptionally small number of your colleagues (often only a single colleague). To run the risk of comical understatement, philosophers don’t always agree. That one editor or a few referees judge your paper to be unworthy of publication is very poor evidence that this view will be the consensus amongst scholars in your area (indeed, I’d very be surprised if we could identify a single paper which every philosopher thinks ought to have been published). Yet, it’s similarly important to remember that the referee is a professional colleague and – at least when things go well – one with a reasonable level of relevant expertise. Often, they will have put a lot of work into their comments, ensuring that – again, all being well – the rejection is accompanied by useful feedback. It’s easy to be critical of referees (fun too), but many of them do an awful lot of diligent work for no real recognition.

So, what’s the best way to respond to rejections? Let’s deal with the easy cases first.

 

How to incentivize better author submissions?

In my past two entries in this series--as well as in other past discussions of peer-review--I have tended to focus on how editorial practices and reviewer behavior might be improved: to better ensure quicker and more reliable turnaround times, as well as better reviewer comments. However, it increasingly appears to me that this is to focus on only one part of the problem.

As David Velleman noted here--and others have noted as well (see below)--another central reason why peer-review has the problems is does is the behavior of authors. Journals are overrun with far too many submissions, many of which seem--to editors and reviewers--as though they should have never been submitted in the first place. This bogs down the entire system, making reviewers not only more difficult to find but also increasing their workload (as they have more papers to review). This in turn may be one significant cause of 'poor reviewer behavior', both in terms of long turnaround times and the quality of their reports. Indeed, there seems to be significant anecdotal evidence for this.

First, respondents to our poll on peer-review generally agreed with the idea that too many half-baked papers are sent out for review:

How to incentivize better peer-review reports?

A while back, I ran and reported the results of an informal survey on what readers think works well and not so well in peer-review. Although the results should be taken with a grain of salt, a number of trends emerged. One was that a vast majority (93.67%) of respondents agreed that 'journal turnaround times are too long and inconsistent.' In a follow-up post, I presented a possible proposal for how to address that issue: a central editorial system (perhaps at Philpapers) for journal editors to use that would give each reviewer a reviewer score, based at least in part on how quickly the reviewer responds to review requests and gets their reviews in. Whenever an editor gets a new paper, the system would then present the editor with the best reviewer matches--the final wrinkle being that the system would be designed to match papers with reviewers whose overall reviewer score is similar to the author's reviewer score. In other words, the system would match authors who are slow reviewers with other slow-reviewers, and authors who are fast reviewers with other fast reviewers.

Although this is obviously a fanciful system, at least at the present time (given that there is no such central system), it seems to me one that might be technologically feasible and work to everyone's advantage: authors, reviewers, and editors (particularly editors, as such a system would vastly simplify their task in finding reviewers!). By matching authors with 'reviewers like them', the system would plausibly incentivize many people to be better, faster reviewers--as there would be some incentive to be a better reviewer (namely, the incentive of getting matched with better reviewers as an author!). I also wonder whether this system might address a second set of issues that respondents in our poll cared about: quality of referee reports. As you can see below, many respondents in our poll thought that too many reviewer reports are incompetently, biased or needlessly aggressive:

Survey results on peer-review

Several days ago, I put up this informal survey to gauge readers' opinions about various features of peer review. I am going to leave the survey up so that people can still take it. But here are the results so far (n = 91), organized by what I will call the good, the ambivalent, and the bad.

As this is only an informal survey, the results should of course be taken with a grain of salt (and then some). Still, I found the results interesting and will be curious to hear readers' reactions.

The Good

64.1% of respondents agreed that anonymized review works well on balance, giving authors a fair chance (whereas only 23.07% disagreed).

An easier poll on peer-review

[Update: survey fixed!]

Following feedback on the poll on peer-review I rolled out the other day, I've tried to design an easier poll. HERE is a new survey that I think readers should have a much easier time with. Although it has 20 items, the items are so simple that I think it should take readers just a couple of minutes to click through. Hopefully we'll get some better data out of this one--and, although it's only an informal poll, my hope is that its results will give us some good material to discuss moving forward!

Reader poll: best & worst parts of peer-review

Based on readers' answers to my query on what works and what doesn't in peer-review, I've put together THIS POLL (poll suspended for revisions - see comments) for readers to rank features of the peer-review process in philosophy from best to worst. Although I'm by no means an expert at putting polls together, my hope is the results will give at least some rough indication of readers' attitudes. We'll see how it goes--and I plan to host discussion threads on specific issues moving forward--but I hope some of you take part!

Peer-review: what works and what doesn't?

Several months from now, I'm scheduled to take part in a discussion on our profession's peer-review practices: what works, what doesn't, and what might be improved. To prepare for the session, I'd like to run a new series here at the Cocoon soliciting readers' opinions on our peer-review practices and norms, and what (if anything) readers think might improve the process.

In today's post, I'd like to ask readers to weigh in on two initial questions:

1. What do you think currently works well in the peer-review process?
2. What do you think the biggest problems are with the peer-review process?

After hearing what readers think, my plan is to run a few informal polls, so as to gauge readers' overall satisfaction with the peer-review process, desire for change, and priorities if change is pursued. Then, after those polls, I hope to have a series of posts discussing some of specific issues in more detail, including more detailed debates about whether the peer-review process should change at all, if so how, and so on. 

So please do considering weighing in on questions (1) and (2). The more readers we hear from, the better discussion we are likely to have in the series overall!