Handling 'split-verdict' R&R's

In our new "how can we help you?" thread, Mercado writes:

I wonder what folks think about the following publishing issue. I just received a second R&R from journal x regarding a particular paper.

This second set of referee comments is mixed. One referee’s comments are on the mark and helpful. But, the second referee’s comments are a bit off the mark, by my lights, and she or he recommends that the editor not publish the paper.

Now, the editor’s communication regarding this second R&R seems fairly positive because he or she uses language like “before we can publish your paper you must respond to these referee comments.”

My question is how much should I countenance or revise in light of the negative referee comments? I ask because seemingly the editor is not taking the second referee’s recommendation to not publish the paper.

In response, Tenured SLAC chair wrote:

This is a fine line to walk. I would say that in the event you do not make changes as per referee #2, that it would be wise to explicitly state why you do not feel they are necessary. This too requires a delicate hand. By the sounds of it, something like what I describe would satisfy the editor. Sorry for being so vague.

I'm curious what everyone else thinks. However, my answer is a bit different. I think what Mercado should do probably depends on what referee #2's comments were actually like. Allow me to explain.

Although Mercado understandably didn't give details about the case, I think there are roughly three possible scenarios here:

Scenario 1 (editors overrule incompetent referee): referee #2's review was so perfunctory and lacking in justification that the editors basically overlooked it in favor of referee #1's positive review.

Scenario 2 (editors find merits in both reviews): the editors found referee #1's positive review good enough for an R&R, but (having read the paper) they agree that reviewer #2's concerns are legitimate and should be dealt with.

Scenario 3 (editors side with good review): both reviews are fairly detailed, the editors found referee #1's positive review good enough for an R&R, but the editors are uncertain about or disagree with the bad review.

I think I've probably dealt with all three scenarios, both as an author and as a reviewer. If Mercado is dealing with the first case--if review #1 is overwhelmingly positive and review #2 is just a few sentences of unsupported rhetoric (e.g. "This paper is poorly argued, but I can't be bothered to explain how!), then the thing to do is to probably just focus on addressing the comments in the good review. In this case, there's really not much to do or say about referee #2's review, and by all indications the editor(s) basically overlooked their review anyway.

On the other hand, Scenarios 2 and 3 are more difficult, and for one basic reason: it's very difficult to know which of the two scenarios one is actually in.

On the one hand, you may think as an author that you are in Scenario 3: that referee #2's detailed review simply gets things wrong. In this sort of case, my experience is that it can be a good idea to simply explain why! In fact, this happened to me as a reviewer once: an author explained how I had misinterpreted something, and I agreed, recommending their paper be accepted after their R&R (as an aside, in this case it also helped that the journal forwarded the other referees' reviews, as their perspectives also helped me re-evaluate my original judgment of the paper).

On the other hand, as an author I tend to think that it may be a better general policy (one which may be overridden in clear cases) to assume that one is probably in Scenario 2 rather than Scenario 3. Here, my reasoning is basically as follows.

First, because as an author I am probably partial to my own work, assuming I know better than a critical referee is epistemically dangerous: I could just be refusing to see a genuine problem where there is one! This actually happened to me a few times early in my career: I felt a referee was just wrong, but later came to realize (after a failed R&R) that they were right. So, with very few exceptions, I've tended to make it a general policy--on epistemic grounds--to defer to referees: that is, to at least address their concerns as "possible objections", even if I personally think the concerns are misguided.

Second, I also tend to adopt this policy for purely practical reasons: both to 'satisfy' the relevant reviewer, but also (and potentially more importantly, I think) for longer-term practical reasons. On the one hand, my general thought is that even if a reviewer is off-base with something, it's probably a better strategy--in terms of actually succeeding in getting the paper published--to address their concerns in the paper, as that at least shows them that you took their concerns seriously and addressed rather than dismissed them. Finally, though--and to me this is generally the decisive thing--I'm inclined to think this is probably a better long-term strategy for the paper itself. For my experience is that if a referee gets hung up on something (even if they are wrong!), other readers of the paper may be likely to get hung up on that thing too after the paper is published. As a general matter I've found that this tends to be the case: that if one reader misreads or uncharitably construes something in a paper, other readers may as well. So, my reasoning goes, better to address it before publication, even if in your opinion the reviewer is mistaken!

But these are just my thoughts. What are yours?

Comments

[Pendaran Roberts]

I have received many split verdicts on my work over the years. I have always worked hard to address all of the comments from all of the referees the best I could and explain why I'm not when I am not. I have written 30 plus page response letters on more than one occasion.

Regardless, I have never convinced a referee who originally rejected the paper to change his mind. Invariably, the referee will make up a whole host of new criticisms (this is most common), or he will ignore my response, or he will claim I somehow did not address his concerns adequately, even though I had. So, based on my experience, I no longer see my response as having the purpose of convincing a reject that the paper is good, but rather I see my response as convincing the editor that the reject is wrong.

However, as far as I can tell, this probably doesn't even matter, because whether the paper will be accepted depends on journal policy. At some journals, they will not accept a paper if any of the referees say to reject. This is especially true at very fancy journals. At other top journals, they will get a third referee and go with the majority opinion.

So, what I'd do, for whatever that's worth, is I'd try to figure out the policy of the journal. Ask the editor if necessary how they intend to handle this split decision. Perhaps they will get another report. But today, given my experience, if I was told I had to change the mind of the rejecting referee, I'd just decline the R&R and send the paper elsewhere. I see no chance of successfully changing the mind of a reject, and I wouldn't waste my time.

[Trevor Hedberg]

When I receive an R&R, I make changes based on all of each reviewer's individual concerns -- regardless of the reviewer's overall verdict -- unless (1) the reviewers contradict one another or (2) the comment relies on a gross misinterpretation of my argument. With regard to (1), I explain which reviewer I am siding with and why. With regard to (2), I explain the misinterpretation and (if possible) alter the manuscript to make the contours of my argument clearer. Thus far, this strategy has been successful for all 6 R&R verdicts I have received.

[Pendaran Roberts]

Trevor, I've only had 2 failed R&Rs. So I've had an 85% success rate. But I've never succeeded in changing the mind of a reject. It's just I've had rejects overturned by other referees, e.g. when 2 out of 3 accept.

Have you ever actually flipped a reject?

[Michel]

I'm in agreement with everyone here: I try to do something about everything, and then explain myself in the letter/changelog. I very occasionally decide not to do anything at all about a suggestion, and I justify that in the letter, too. In one case, I was pretty forceful about a few concerns that I thought were misapplied, and it turned out well.

But: Pendaran, you sometimes have 30+ page changelogs? O_o

[Pendaran Roberts]

Micheal,

Yes, I've easily had 30 pages and more for my response letter.

I've never flipped a reject, but I have on more than one occasion had the accepts comment on the reject's report and side with me.

I think that helped but was probably still just journal policy that decided the issue in the end.

[Amanda]

I agree that if a reviewer says reject, it is low odds you will change their mind.

As much as possible I always try to respond to every reviewer comment, and see if I can at least make some changes. I would guess my success rate is something like 75%? I might be biased, but I think I have had some particularly unfair cases. I had two awful cases that stick out. One had 5 rounds of revisions over 1.5 years and an eventual rejection. Each round the reviewer got more and more aggressive, and for some reason the editor kept encouraging me to do the revisions only to reject it in the end. It was awful. Another time I wrote 30 plus pages to have it rejected. What I think happened in that case is the reviewer (only one!) didn't really want a RandR but the editor gave it to me anyway, which turned out to be a waste of time because the reviewer wasn't going to change his/her mind.

[Dan]

30+ pages strikes me as terribly long and unduly burdensome for reviewers, though as I'm new to the publishing world, I may of course be ignorant. Is it customary for response letters to be that long?

[Pendaran Roberts]

"One had 5 rounds of revisions over 1.5 years and an eventual rejection. Each round the reviewer got more and more aggressive, and for some reason the editor kept encouraging me to do the revisions only to reject it in the end. It was awful. Another time I wrote 30 plus pages to have it rejected."

The 1.5 year situation is absurd. I'd never submit to that journal again and would have probably gotten quite angry with the editor. I'm really sorry you went through that, and I think it's just immoral to put someone through such an ordeal to just reject them anyway in the end.

My worst experience was with AJP where I had one reviewer who wanted the paper expanded significantly and unreasonably in my opinion. The editor seemed to understand that what the referee wanted was over the top and said explicitly not to expand the submission by very much. There were three reviewers. Two accepted my revision, one even saying very positive things about the paper, but of course the one who wanted it expanded a lot was not happy that we hadn't and rejected it. The editor didn't care that he had said not to expand the paper very much and rejected anyway. I was not happy and said as much. I won't submit there again.

"30+ pages strikes me as terribly long and unduly burdensome for reviewers, though as I'm new to the publishing world, I may of course be ignorant. Is it customary for response letters to be that long?"

In my experience yes, but keep in mind that about half of it for me is quoting the referees' comments so I can respond to them.

[Amanda]

Dan, no I don't think it is customary. And usually I don't recommend doing so. I think you are right that in most cases it would burden the reviewer and not really help your case. The one time I did it there was no other way to respond to the reviewer's comments. Alas, the one time I did that it wasn't successful either :(

[Referee]

Dan,
I would not write 30 pages in response to referees' reports in an effort to submit a revise and resubmit. At most, I would write 2 pages. Just politely acknowledge the constructive feedback from the referees. Note how the paper has been improved. (Be sincere) Then address the remaining critical remarks, either explaining why you did not address them, or why you do not think they are salient but addressed them anyway, either in the text or in footnotes. Thanks the editor for reconsidering your manuscript, and press SUBMIT.

[Marcus Arvan]

I too think 30 pages is probably far too much (though perhaps Pendaran has been successful with this). On the other hand, I'm not sure I agree with Referee's suggestion of writing 2 pages at most. I think it entirely depends on how many referees there are (in some cases there are 3!), how many problems they raise, and so on. Looking back at some of my successful R&R's, many of my author-responses are somewhere between 3 and 10 pages.

[Pendaran Roberts]

Hey Marcus,

I have an 85% success rate with R&Rs. I'm not sure what my success rate is for R&R's that involve very long response letters, but I don't think it's worse. Obviously the longest response letters occur when I have 3 referees.

I've always taken the response letter very seriously and that has at least not seemed to backfire for me. But I had no idea some people just write 2 pages. I would never have dreamed of doing that. Maybe I should start!!!!! haha!!

[Pendaran Roberts]

Folks, keep in mind that when I say 30 pages that includes quoting the referee. So, there is only half that amount of space for my comments.

Is this really that unusual for submissions with 2-3 referees???

I didn't think so...

[Amanda]

As said, 30 pages is typically too long. But a two page limit strikes me as way too short.

[Referee]

I remain committed to the two page reply to a revise and resubmit. In fact, I have only once been ask to r&r and then had it rejected (SEE BELOW ... it gets exciting). I do treat the referees' comments with care - and I bundle them. "Ref1 and Ref2 both raise concern X..." ... and then I explain what I did in a sentence, and where it can be found in the revised paper (see page 7). Once an editor did reject a revised manuscript, but I fought them on it. This was a very good journal. I was in a contingent job, a few years after my PhD, so it took some nerve. But I argued that they had been very long with the paper, and I had made the changes. This is some time ago. I am glad I did. The paper has been cited over 100 times.

[Michel]

Ah, it makes more sense to me if it's 3 referees and you're quoting. As far as I know, I've never had three referees at once.

I don't actually quote the referees in my changelogs. I try to summarize their comments instead, and divide them between bigger and smaller concerns. So they're usually around 3-5 pages long.

[T]

A 30 page changelog is exceptionally out of the ordinary, quoting or not. Hell, a 30 page paper is out of the ordinary. I'll be blunt: if I refereed a paper and it came back with a 30 page changelog, I'd be liable to reject it on the grounds of not wanting to read the damn changelog. Of course, I'd *say* I was rejecting it for some other reason...

[Pendaran Roberts]

Having thought about my practices, it occurred to me that I learned them from a notable psychology professor. My method has the obvious advantage of making it unquestionable that I've dealt with every referee comment. I don't summarise or combine or anything: Every comment has a response. I guess the negative is that my response letters are long. However, they've had a good success rate for me, so I plan to continue my approach.

[Untenured Ethicist]

From a reviewer's perspective, the best response letter is whatever makes it easiest to determine whether the new manuscript responds well to the referee reports. A shorter letter takes less time to read. But some things that make a letter longer, such as signposting, quoting the review, or being more explicit about what has changed, may save the reviewer time overall.

Regarding Mercado's question, I would take the editor's word about what changes are required. If the editor said, "Before we can publish your paper you must respond to these referee comments," then you must respond to those comments. It's possible that you could persuade the editor by responding only in the letter, explaining why the reviewer is mistaken. But it's unlikely. Better to change the paper in a way that enables you to truthfully say you've addressed the reviewer's worry.

Telling reviewers they are wrong is asking to get rejected. Even if the reviewers are, in fact, wrong.