Our books






Become a Fan

« Why "anonymized" review isn't, and what to do about it | Main | Does skilled perception confer justified beliefs? »

07/21/2015

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Wesley Buckwalter

Several months to get desked without comments is so outrageous to me, I don't know why we tolerate it. But I think we disagree about how helpful referee reports typically are to authors or how much they would improve papers if implemented. IMO sending papers to journals seems like one of the most ineffective ways to get actionable comments on your work ever. On the other hand, perhaps an online community like Cocoon could help with prebublication review for junior members in the areas contributors are interested, and I suspect it already has with some of your initiatives.

Sam Yelnik

Honestly, I do not see it as my job as a reviewer to put much effort into improving somebody's paper.

Michel X.

One small possibility that might be operative in some cases, but doesn't really change anything: perhaps the editor deemed the referee's comments not worth sharing (e.g. perhaps they were too vitriolic but the complaints seemed legitimate). I guess in that case I'd expect the editor(s) to distill *something* from the referee's comments, but maybe that's a naïve expectation on my part.

Helen De Cruz

Wesley: I have had a mixed bag of reviews, I am thinking of people who have little means of having quality feedback on their papers (such as at small departments or with limited funding for travel). Your mention of online media is something I hadn't thought of (which is funny since I sometimes testrun new ideas on blogs and academia.edu) - definitely something for philosophers with limited other opportunities to receive feedback. I would also take this opportunity to invite anyone who wants to be the next featured early-career author to contact Marcus Arvan to see their work featured on the Cocoon.

Helen De Cruz

Sam Yelnik: I think this depends on the referee. Personally, I recommend reject in the large majority of cases but I still offer suggestions for improvement. We know that most papers are not accepted upon first submission, and it seems to me that providing feedback is one way of shortening the journey of a paper. I know some people just don't take on any referee comments at all. However, I have had a paper sent to me that I rejected earlier for a different journal. If none of my comments are incorporated, that is of course at the author's discretion, but it does not make for a favorable read of the paper.

Wesley Buckwalter

@Helen Yes it is certainly a mixed bag. So much so that I would recommend people with limited opportunities to literally pursue any other strategy first to receive feedback on work in development before turning to journals reviewers.

Scott Clifton

Personally, I think that rejecting a paper without giving, or being required to give, some sort of rationale is intellectually lazy. I recognize the value that good feedback has for authors, but I think the primary value of requiring some feedback from reviewers lies in the reviewer's having to justify the decision. If this were the standard, then the set of comments could be short and aimed squarely at the reasons for the decision, even if some reviewers wanted to aim also at helping the author make the paper better. Nevertheless, these are two distinct reasons for being required to give feedback and, like Sam Yelnick, I don't think it's an obligation for reviewers to try to make the papers better. They should still justify their decisions, however, with some sort of feedback.

And regarding desk rejections: it does seem to me that I am getting more decisions that seem to be desk rejections, as there is no indication that the papers have been sent to reviewers, and these are coming several weeks after I submit them. I have always thought that the fact that desk rejections open authors up to greater chances of implicit bias, as well as easy, unjustified rejections, is counterbalanced by the shorter review time generally found in desk rejections. Therefore, as some journals are now giving desk rejections weeks and sometimes months after submission, there is NO value to authors in desk rejections. It's now only an advantage to editors, who have almost all the advantages already. So yeah--shitty trend, if it is a trend.

Neil

Marcus, "under review" doesn't always mean what it says. At least in some versions of editorial manager (but not others), papers are marked "under review" as soon as an email to a potential referee is sent. It then remains "under review" until two referee reports are received. So a paper may be marked "under review" while multiple people ignore multiple requests (and the one actually takes the time to decline).

Roman

I am going to chime in and say that referee comments are crucial, even in cases where I've already presented a paper at a conference. A careful reading often brings out a number of problems that conference attendees didn't catch. Also, referees tend to be much less sympathetic than conference audiences, and I like that: I need to know what the major problems with my paper are (also, I can sometimes be a decent presenter, and I worry that audiences might be favorably swayed by features irrelevant to the paper's actual quality). Reviewers don't need to do work to improve my paper, but I would appreciate knowing what their reasons were for rejecting it because chances are I'm going to try to get it past other reviewers, and if it has specific problems, a desk rejection isn't helpful.

Marcus Arvan

Sam Yelnick: You write, 'Honestly, I do not see it as my job as a reviewer to put much effort into improving somebody's paper.'

In my experience, yours is a common view in philosophy. I've heard it from many people. But I do not think it is a good one, and it may be helpful to compare our field with others. Here is the "editorial policy on reviewing" for Cognition, a leading journal in psychology: http://bpa.ac.uk/uploads/Good%20Practice%20Scheme/Cognition%20reviewing%20policy.pdf

Yes, the journal's official policy is that, "Reviewers have a responsibility both to the science and to the authors who are trying to advance that science. This responsibility includes helping the author better his/her paper and, if necessary, better his/her science...The role of the editors and the reviewers is as much to reach a consensus on how the author could improve the impact of their research as it is to reach a consensus on whether a paper should be accepted."

We could learn from other fields. The idea that "it is not our job to help authors out" seems to me one that does not reflect well on us, or on our discipline.

My wife works in psychology, and in my view their approaching to engaging each other's work is so much more supportive and productive than ours.

Marcus Arvan

Hi Wesley: I have to disagree. Although I have received my fair share of desk-rejections and irresponsible reviewer comments, a good amount of reports I have received have been *invaluable* in revising papers.

In my view, this very fact--the fact that conscientious, detailed review comments can be so helpful--is a reason why they should be an editorial expectation.

Marcus Arvan

Hi Neil: I think you might have misunderstood this post as being written by me. Helen wrote it! :) (Maybe we should start off our posts by indicating who the author is)

Roman

I've never received a rejection with helpful comments (because I've only received one rejection with comments; the rest were desk rejections), but my R&Rs have been extremely helpful. Even where I think they are completely misguided, they show me how a sloppy reader might completely misconstrue my point, and this is valuable knowledge. At the very least, almost any set of comments will convey some important pieces of information: (1) Did they paper make sense? (2) Are there obvious gaps that need to be filled? and (3) What did a reader *think* I was arguing? Desk rejections convey no information. At most, they say, "This paper did not sufficiently advance the field." That's not information, since sometimes it is patently false, relative to the journal's other output, and it's obvious boilerplate.

Wesley Buckwalter

@Marcus I agree that this should be an editorial expectation.

Sara L. Uckelman

I thought by definition a desk rejection was when the editor rejects a paper without having it reviewed. Thus, if it's gone to the reviewers, it's not a desk rejection, even if the reviewers don't provide any comments that the editor chooses to forward.

Sara L. Uckelman

I recently received an interesting referee request: "Note that we have adopted a new procedure designed to facilitate the work of referees. We ask you to read the paper within 2 weeks and send us an initial reaction. Your initial reaction may be a 'summary judgment' accompanied by a brief explanation. Alternatively, you may decide that the paper merits more intensive consideration and comment, in which case you should specify your own deadline, within 6 additional weeks, for submitting a full report."

While I can understand wanting to try out a new system which decreases turn around time, my suspicion is that it will do so at the cost of an increase in rejections, probably with very little comment to forward to the author, because unless the paper is acceptable as submitted, the only option if one doesn't want to commit to a more in-depth process is to reject.

I opted for the first course, but am still ambivalent as to whether I should have, or whether I think this method of soliciting reports is ultimately a good one.

Helen De Cruz

Interesting experience, Sara Uckelman. I have not yet encountered this system as a referee (I've always been asked for comments). I do think it is equivalent to a desk rejection (although, technically, you are right only things rejected from the editor's desk count as desk rejections).

Sam Duncan

Here's another case of editors behaving badly that I'd like to see what people think about: On two different occasions I've had editors decide to reject contrary to the referees' recommendations. In one case the referee recommended conditional acceptance but the editor rejected. In the other one referee recommended accepting the paper as it was and the other recommended significant revisions and the editor rejected. (These were two different journals. One's a generalist journal and other is an applied ethics journal.) Has anyone else run into this? I think it's really lousy behavior on the part of an editor. For one thing, doesn't rejecting contrary to the referees comments make a mockery of blind review? Worse, to reject in these cases I think the editor must have pretty much decided that he thought the paper should be rejected before sending it out. But if that's the case why send it out in the first place? Why waste my time and the time of the referees? That's precisely when he should desk reject a paper.

recent grad

Sam,

It may be that your paper is in the running for a limited number of R&R and acceptance spots. Even if the referees think it deserves a spot, their recommendations may not be as glowing as the recommendations for other papers at the same journal. It's a zero-sum game, after all. This explanation is consistent with the editor *not* being against your paper from the start.

All that having been said, I agree that it stinks. It's happened to me too.

Calvin

Hi Sam^: I agree that this behavior seems suspect, and I have never been an editor for a journal, but I used to be on the editorial staff at a university press. My experience there was that desk rejections are rare, mainly because authors self select pretty well, and that when a manuscript was reviewed the editorial committee almost always followed their recommendation. On occasion, they did not. This was generally not for academic reasons, but for budgetary reasons. I'm not sure how many of these worries might extend to journals, but at the very least I could see how a journal could reject an otherwise excellent article simply because they didn't have room for it, or they felt it was too much like another forthcoming article that the reviewers were not aware of.

perplexed editor

Sam,
The behavior of editor you describe does not sound unreasonable, at least not in the second case. Remember, the acceptance rate for many philosophy journals is quite low (10% - 5 %). And it is likely that a number of papers come in with two very positive reviews. Other papers will come in with one positive review, and a recommendation for MINOR revisions, rather than significant revisions. All these papers are going to be more deserving of a space in the journal before your paper as described. What you should do is revise your paper in light of the comments and send it to another journal. The fact that you did get a positive review suggests that it stands a decent chance of getting published sometime.

Neil

On the request reported by Sara: summary judgment within 2 weeks or full review within 8. I have got those requests, and I always provide a full review within 10 days (usually much more quickly). For most of us, that's just good time management, not an impossible demand. Face it: if you delay 3 (or 4 or 5 or however long you usually wait) because you're so busy right now, you end up doing it - eventually, grudgingly - at some other time when you're just as busy. Next time you're looking for something, anything, to do rather than referee that paper, google the planning fallacy.

Roman

"Perplexed": That's interesting. It still sounds unreasonable to me. Let's say you, as editor, have chosen two reviewers, A and B. A is very positive about the paper, while B recommends significant revisions. Clearly something has gone wrong. Maybe A or B isn't particularly competent to make such a judgment. Maybe A or B was lazy. Maybe he paper made a very interesting novel point that was over B's head. Of course you could say "Well, I trust B." Fine. But if you reject in this case, then you're not really trusting B, and you're completely not trusting A. I get if the editor just has too much to do and rejects in order to avoid figuring it out further. But if that's the kind of rationality by which journal reviews are edited, it seems antithetical to what that goal of journals is supposed to be in the first place (of course that is where we are--many journals operate as vehicles for reproducing the same thing over and over because, frankly, that's what our current publish or perish model requires, as it's impossible for this may people to produce a constant stream of original material; but is it the kind of thing we should defend as "not unreasonable"?).

Neil

Is perplexed claiming that top journals only publish papers that get no worse than minor revisions verdicts from referees? That's very clearly false. In my experience, as an editor, author and referee, straight acceptances and minor revisions are very rare. I have received r&rs thar led to publication and I have issued r&rs that lead to publication. In fact, almost all the papers I have published and almost all I have refereed that were subsequently published got r&rs.

perplexed editor

Neil, Perplexed is not "claiming that top journals only publish papers that get no worse than minor revisions verdicts from referees".
What I am suggesting is that (i) Sam and others should not be surprised when a manuscript is rejected when it receives a verdict of significant revisions; and (ii) Sam should look to another journal, and not get bogged down on bad feels and thoughts that he or she is not in a position to verify.
Roman also misunderstands my point, and the way peer review works. It is an imperfect system. But good papers will get published. There are enough journals out there.
I am rushing and fear that I may be leaving others with more perplexing thoughts on this.

Scott Clifton

If the journal is that endowed with excellent papers in the pipeline, then maybe the reviewing standards themselves need to be adjusted. What would have been a conditional acceptance or R&R would now just be a rejection. Maybe it doesn't matter substantively where the hard decision is supposed to be made--at the reviewing or post-reviewing stage--but for the sake of authors who are probably unavoidably epistemically limited, it might be better to have reviewers outright reject, though with the same set of comments the author would have received prior to the adjustment. Of course, this could lead to head-scratching on the author's part--"Huh, the reviewer had almost nothing but positive comments--why was the paper rejected then?" Is this better than "Huh, the reviewers seemed pretty pleased with the paper--why did the editor reject it then?"?

Neil

"What I am suggesting is that (i) Sam and others should not be surprised when a manuscript is rejected when it receives a verdict of significant revisions".

No Perplexed, you didn't just say that Sam shouldn't be surprised in these circumstances. You gave reasons why the verdict shouldn't be surprising: that papers with reports like Sam's will often (often, because otherwise the situation described would be surprising) by papers with better reports. And in response I claimed that I doubted that this was the case: the vast majority of accepted papers receive r&rs, on the basis of reports that recommend r&rs, and that the situation you describe is rare enough to count as surprising. Our dispute is not whether Sam should be surprised, but about the explanation you offer for why he shouldn't be surprised.

recent grad

Neil,

I think perplexed point is the same even if it's not straight acceptances/minor R&Rs that are getting in over significant R&Rs. All that is required is that some R&Rs are seen as more promising than others, whether for their intrinsic content or for how well they mix with other papers to be published.

Neil

Sure some r&rs are more promising than others. But the initial cases were a conditional accept and a straight accept coupled with an r&r. Few - very, very few - verdicts are more promising than that.

Sam Duncan

A lot of this could be avoided if journals gave referees and editors detailed guidelines. If you have too many papers in the pipeline then make it clear to the referees that they need to be very very strict, and that only something really original and tightly argued should even get an R and R. By giving editors as much discretion as they do a lot of journals make a mockery out of blind review: The editor, who knows who you are unless the journal is double blinded, can do pretty much what he feels like with your paper no matter what anyone else says. Also factor into account that the editor almost certainly knows a lot less about the subject of your paper than the referees do and letting them go counter to the referees looks even worse. And "surprised" is not how I'd describe my reaction by the way. "Outraged" or "disgusted" come a lot closer to the mark.

Kenny

I agree with several earlier commenters that it is very important for referees to justify their decision. When I'm grading papers, I'm thinking to myself, 'what would have to be different for this to be an A?', and when I'm refereeing I'm thinking, 'what would have to be different for this to be an outright accept?' In both cases, I try to communicate that to the author. And that's what I expect (in the normative, not descriptive, sense of 'expect') from those who are reviewing my papers. Now, explaining what would have to be different is NOT the same as explaining what revisions would improve the paper to that point: sometimes a particular paper has no hope. Other times, the referee can't see how the paper could possibly be fixed, but the author might. But you still have to explain why. So this isn't necessarily the same as saying the referee has a duty to improve the paper; I don't disagree with that point, but I think this point about explaining why the paper's not accepted is more important.

Related to this: of course it's frustrating and discouraging when you get really nasty referee reports, but it's also frustrating when you get a referee recommending a negative verdict and you can't see how the referee's specific comments justify that verdict! In other words, I have occasionally seen cases where the referee is too 'nice' in the comments and thereby fails to make their verdict intelligible.

Some journals do provide a justification for desk rejections. Journal of the History of Philosophy, for instance, has a very fast turnaround time and issues a lot of desk rejections, but they're triple blind and they usually include a one paragraph explanation of why the editorial committee decided not to send the paper to review. This is a great practice.

I've also heard of (but never personally encountered) some journals having a list of acceptable reasons for referees to give a summary rejection verdict (i.e., rejection without comments), so that the referee can check a box with their reason, and unless their reason fits into one of those categories they're expected to write comments. This also seems like a good practice to me, as long as the checklist is well designed.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)

Job-market reporting thread

Current Job-Market Discussion Thread

Categories