Refereeing practices for better journal publication experiences

I was recently talking to a philosopher who had to wait 17 months for an initial decision on her paper at a top journal. Another had to wait over 12 months and has now retracted her paper after the editor remained unresponsive to her inquiries on the status of her paper. In both cases, the slow journals were in the top 5. I have had many discussion with people about how awful reviewers are, how they systematically misunderstand the points we make, or try to get citations to their own work into our papers.

Here's a puzzle: Why do so many people (me included) think they are conscientious, timely referees with constructive feedback, yet do we end up having our papers reviewed by tardy, careless, uncharitable referees? Since we have no reason to assume we're special, something is going on. It might be that we overestimate our own quickness and constructiveness. Also, we might underestimate the value of reviewer comments we receive. This seems likely given that me-side bias (the bias whereby we systematically underestimate the strength of views not in favor of our prior beliefs) is not attenuated in highly educated people. 

Looking over my CV in the section "services to the profession" (which is pretty useless for career purposes), I see that I have reviewed for at least 24 journals (not all of them in philosophy), and 3 academic book publishers. Most of these requests have come in over the past few years. I have adopted a set of practices that I find useful in keeping reviewing manageable and that hopefully maximizes the benefit my reviewees get (see also here, Thom Brooks's guidelines, some of which converge with the practices outlined here)

1. When I get a request to review a paper, I accept or decline the request on the same day. When declining, I try give names of alternative reviewers. Apparently, many editors have to wait up to a week for a decision on this; in this way, valuable time is lost, especially if the hesistant prospective referee declines, and one needs to find someone else. I think this might be because we put off the decision (not wanting to be uncollegial, but not wanting to do it either).

2. To aid my decision, I immediately decline if
   2a. I don't have any time to do the review in the next 4 weeks.
   2b. I already have a review to perform, i.e., no more than one review task at the same time.
   2c. The paper is on something I would need to do extra research on to give an informed decision on its merits. This seems also fair to the author. So even if it's a paper in my AOS (philosophy of cognitive science), if it's something I'm not up to speed about, I say no. 
   2d. I am certain who the author is. I think we are less good at guessing authors' identities than we think, therefore a suspicion is not enough. With certain I mean, for instance, that I saw the author present the paper. Even in those instances, editors have asked me to review anyway since it is a very small field and I am one of the few people who could give a verdict. If that is the case, I accept the review request since I guess it's worse not finding any referees for your paper than having someone review your work who knows who you are.
   2e. For commercial publishers that impose heavy embargoes and demand high subscription prices for journal packages, like Springer and Elsevier, I cap my reviews at 3 per year per publisher. I'd love to support open access journals (unfortunately, that's not for me to choose).

3. The anonymity of the author: There has recently been discussion on unethical practices of referees (e.g., Googling titles) that might aggravate biases against some authors, especially junior people, people from smaller institutions, women and other minorities. Even when you don't google the title, the anonymity of the author is not guaranteed. When reviewing a paper, all sorts of ideas about the author involuntarily pop into my mind. I of course do not try to find out the author's identity but I found myself thinking, one of the last times I reviewed a paper, "Is the author a graduate student?" This is very troubling, since I would not consciously take this as a reason to reject a paper. But implicit biases are legion. For instance, if one guesses a paper might be written by a grad student because of its very lengthy and thorough review of the literature (older authors, so is my impression, are more succinct in their attribution of sources). Or perhaps the language suggests the author isn't a native English speaker.  So what to do here? I have no easy answers. I try to be aware of such thoughts, especially on how they might affect my decision process, and focus on the quality of the paper, but I am not sure what additional steps I could take. Hopefully Cocooners have useful suggestions? I think implicit bias is a huge problem in philosophy. 

4. My own anonymity: I usually remain anonymous, given that rejection is the most common decision and one doesn't want to make enemies for life. A few years ago, I refereed for Frontiers, which has the policy of publishing the names of referees alongside accepted papers - this way, the author finds out who the referees are *if the paper is accepted*. This is very good policy - if you know your name might appear eventually, it makes one more careful as reviewer. So I've recently taken to signing my name under comments when the paper is at its second round of revisions, i.e., when I get revisions back from author, and provide my thoughts on the revised version.  I hope by revealing my identity to the author at some point in the process I would do a better job at reviewing.

5. What is publishable? I have recently adopted a policy of positively recommending papers that make an original point, even if there are some flaws, as long as the overall structure of the argument is sound. By contrast, I tend to recommend rejection for papers that seem to meticulously pre-empty any concerns but that only have an incremental value in the debate they are situated in. Journal publishing is after all a zero-sum game, and in spite of low acceptance rates, I find there is too much incremental stuff in the major philosophy journals. There is scope for disagreement here, but reviewing gives one the opportunity to put forward what one thinks are important desiderata for the profession (and I prize innovation over sudoku-philosophy).

6. Reviewing time. I try to keep reviewing time under 4 weeks, but it is unfortunately not always feasible. So I take the journal's allotted review time as a final deadline for a paper to review. When my own writing goes well, I prioritize that. If I need to give it a rest, reviewing is a nice way to concentrate on something else (it's not the same as grading student papers or reading papers for research, and it feels like one has done something substantial at the end of the day).

7. Communicating a verdict: Years ago, I made the mistake of being invariably too nice in my reviewer reports (inspired by my own reactions to devastating review reports). So I reserved my most devastating objections to the confidential comments to the editor. The problem then was that authors got a report with lots of constructive feedback but no clear reason for why their paper was rejected. So I now try to avoid confidential comments to editors (exceptions would be if I found evidence of plagiarism or the like), in this way the author has a clear idea of my assessment of strengths and weaknesses of the paper. 

 

Comments

[Neil Levy]

Very useful set of guidelines, Helen. I keep records of my reviewing, so I know I am not wrong in thinking that my review time is much quicker than others seem to be (I have never taken more than 10 days to referee a paper and the mean is under 7 days). I do wonder whether there isn't some confirmation bias plus some kind of saliency bias in the idea that the turn around times afflict the top journals especially. My worst experiences have come with specialist journals, not the top general journals (again, I keep records). Two specialist journals have taken more than four *years* to return an initial verdict on submissions. Of course submissions to the top journals will be especially memorable to us, and that might partially explain why bad experiences with them are especially salient.

[Rachel]

I tend only to write confidential comments to the editor if I want to be either more harsh or more glowing in my recommendation.

E.g., if a paper is just *terrible* and I think that under no circumstances should the journal publish it (even as an R+R), I'll say that in my comments to the editor. To the author, I'll say something more kind like, "In my opinion, this paper is not publishable in its current format, and I think it would take such significant restructuring as to essentially constitute a new paper."

E.g., if I think that a paper really ought to be published, to an editor I may write: "The journal would be extremely well served to publish this paper. I can't recommend it highly enough. It's fantastic."

[Helen De Cruz]

Rachel, that seems very sensible. It's important to let the editor know you really like a paper, given that the default response is rejection (given space considerations), and sometimes it's important to convey that a paper will likely not reach the standards of the journal even with a revision. One reason I tend to try to avoid comments to editor is that I came off as too nice - giving helpful suggestions and pointing out the strong points of an article, when I thought overall it was weak.

[hgi]

I do not think that reviewers should ever send separate comments to the editor. We are being more transparent, despite the anonymity of peer review, if we give the author exactly what we give the editor. Also, there is something slightly perverse in sending a harsher comment to the editor. The author then does not know how bad their paper is.
I also do not understand the long times for refereeing. I referee about 12 to 16 manuscripts a year, in addition to other editorial work, and many of the journals use on-line systems. The systems tells me I get my reports in on average in 1-3 days. I take 10 days for long paper, and almost always regret that I agreed to referee it (by long I mean 50+ pages).
Helen's rule of never accepted a request while one has another paper on one's desk is an excellent rule.

[Helen De Cruz]

HGI: these are also my reasons for avoiding comments to the editor. Perhaps one can make a strong recommendation that the author can see (as in Rachel's example). It must be frustrating if the paper then still gets rejected, but it's good to know that at least someone who is your peer really liked it. Conversely, it's important that the author of a really subpar paper realizes that it would require a lot more work (as a new submission), so that's why I am now somewhat more direct about the shortcomings of a paper than I used to be - one can do that without being nasty, I think, e.g., saying the paper doesn't show sufficient familiarity with the literature, where the argument went wrong.

[Marcus Arvan]

Sorry if this is a tangent, but I have a question that hasn't been discussed yet. Suppose you've agreed to referee a paper, but then, while reviewing the paper, you become quite certain you know the author of the paper (e.g. they refer anonymously to other stuff they've published, but you *know* who wrote those papers). Should you let the editor know and/or back out of the review? Or, especially if you've had the paper for a bit, should you still do the review? Thanks!

[Helen De Cruz]

Hi Marcus: I haven't been in that situation yet, but my thoughts are that you should let the editor know that upon reviewing the paper, you came to know who the author is, and ask them whether they'd still like to complete the review. It isn't ideal, but my guess is that many editors would go ahead and ask you to do the review anyway (based on my experiences where I knew right away who the author was, because the field is so small and I know their work or saw them present). However, if there's a potential conflict of interest, for instance, if the person is a good friend, I'd say to the editor that I know who the person is (through things they reference), and since I know the author personally, I would be unable to provide a fair review.

[Christopher Gibilisco]

I've made a quick list of precautions I used on my website to keep referees from searching my paper titles. I hope other junior philosophers find it helpful.

http://www.cjkgibilisco.org/index.php?id=15

[Tom Mulherin]

Christopher (if I may), that link seems to only go to your home page. When I click through to research and click on the link in the note to your papers, it kicks me back to the homepage again.

[Christopher Gibilisco]

Tom: thank you, and sorry for the slow reply. I've fixed the link. My host likes to kick people to the homepage instead of a 404 page.

[ross]

Thanks for the post! The guidelines for helping one decide whether/when to accept a refereeing duty are helpful.

However, there is rarely good reason to not recommend that an editor accept for publication the paper that one has agreed to referee.

The most pressing question before the referee is: Are my concerns about (a) the argument/view/etc. presented in the paper or (b) the way the paper is written or (c) etc. important enough to make it more difficult for the person who authored it to find employment where having a publication is highly important? For most, it would be surprising if the answer weren't, 'No'. Furthermore, typically it would require an entire paper or two for a referee to substantiate the conclusion that the paper should not be published on the basis of their concerns.

Hence, it is upsetting to hear that someone mostly recommends against publication. It is especially upsetting to hear that that someone mostly recommends against publication on the basis of whether they think the author's central idea and/or argument for it is novel as opposed to whether author's argument/arguments for the central idea are very good/plausible/etc. fml.

In any case, whatever.