Why three reviewers?

In our newest "how can we help you?" thread, a reader asks:

I have submitted a paper to a special issue of Synthese, and it is now under review of the third reviewer. Is it normal to get a paper reviewed by more than two reviewers (in my case, three) in a special issue? Does this mean that there are conflicting suggestions from the first two reviewers?

Another reader, "ref", responded:

Yes, there probably was a split verdict, but one was promising enough to warrant a 3rd opinion

This seems probable to me, but I wonder whether journals sometimes commission three reviewers even without a split verdict. Any editors care to weigh in?

Comments

[Michel]

Another possibility is that the editor of the issue is being labelled by the system as one of the reviewers. (I don't know, it's just speculation.)

[anon]

Might possibly have some relevant info to offer. I've refereed as a third reviewer for a manuscript that had already received two positive reviews (of course, I only learned this after that fact, i.e. after submitting my review.) Why the editors sought another reviewer was not clear to me, though I ended up asking for major revisions.

In addition, I've been on the other end too. I currently have a paper that's been under review for over 3 months. Just recently it moved to the status "reviews completed", but then several days later it was sent back out and is now under review again. It's annoying to be sure, particularly as an early career person applying for jobs right now where having a paper accepted sooner rather than later could make a real difference. Also happens to be for a Synthese special issue.

[Nick]

I've anecdotal evidence that editors will look for a third referee if the two verdicts are not unambiguously positive and worth relying upon. This can be very frustrating when it seems like the editor is fishing for reasons to reject.

[anonymous associate professor]

the special issues (now called "topical collections" at Synthese require that both referees recommend acceptance (this can be after rounds of R&R) for a paper to be published (this is one of the things they did to address their (maybe two different?) scandals with special issues a while back. So I would be surprised if the guest editor was allowed to send it to a third reviewer if one of the referees recommended rejection. However, I could see a scenario in which the rejection report was so shoddy or irresponsible or nasty or something like that that a case was made that it should be re-reviewed. I believe their policies about this are on their website (look around for information for guest editors and you will find it). Is it possible one of the initial two reviewers just didn't submit a report, and said that they couldn't, so they ended up looking for another one? (That is, are you sure there are two extant reports?) Another possibility I guess is that both referees recommended acceptance and the guest editor did not like your paper so intentionally sought out a third. I'm not sure they are allowed to do that either though.

[anonymous associate professor and associate editor]

Sometimes a paper that spans two sub-disciplines requires a third referee, given the trends toward increased specialisation within philosophy. It's happened to me, i.e. as an author, where (a) the paper was controversial within my own specialisation, requiring two referees within that area, and (b) crossed into another specialisation, requiring a third referee.

It's also happened in my role as an Associate Editor for a top-tier journal: in this case it was a paper on Kant that was also working at the intersection of contemporary metaphysics and philosophy of science. So, two referees for the historical side, and one person in the contemporary metaphysics stuff who was conversant with the historical stuff.

[EuroProf]

I have occasionally asked for an additional review when the original reviews were either not helpful (e.g. 'this is a great paper, I enjoyed reading it, accept') or gave vastly divergent opinions (e.g. reject-accept).

[Synthese TC editor]

anonymous associate professor has it pretty much right. Where Synthese submissions receive two very polarised reports, they sometimes seek a third referee. Are you certain that is what's happenned in your case? For me, Editorial Manager just says "under review" or "referee's assigned." It doesn't say how many referees are assigned or how many have been approached, etc.

[R3 H8s Me]

Yeah this is all incredibly confusing. I recently had a paper where R1 asked for minimal revisions and R2 asked for zero revisions. They then sent the paper to a 3rd reviewer who asked for pretty substantive revisions.

[R4]

R3 ...
sometimes R3 is invited because R1 or R2 has not returned their report in a timely fashion - even though they accepted the invitation. But then, after R3 is invited, R2 gets their report in. So the editor ends up with 3 reports.

[just trying to understand]

Thanks for all who are weighing in. What I'd be curious to know is whether there are semi-standard practices regarding all of this, or whether it simply varies editor to editor, and journal to journal?

For example, as in the case anonymous ass. prof. and ass. editor mentioned above, if three refs. are needed because the paper topic demands it, would three referees then be sought out from the onset? Or, might an editor ask just one ref. at a time, waiting for each report before asking for another? Likewise, can an editor reject a paper even when it gets positive reviews, and/or can they send a paper out again even after it receives fairly negative reviews (e.g. a reject and a major revisions?)

I'd really like it if journals/editors presented their standard practices somewhere (or at least made it explicit that they don't have any!) as it would have a big impact on where I submit manuscripts.

[anonymous associate professor and associate editor]

At the journal that I am an associate editor (AE) for, it is standard practice to begin by setting up one referee to review the paper; if that report is reasonably positive (doesn't have to be outright accept, which is very rare, but an encouraging R&R verdict) then we go to a second referee. It can be very difficult to find good referees; it is dispiriting how many people simply do not respond to invitations to referee, or have a track record of not accepting a single one. So, it is a resource to be managed carefully: hence the above practice. Moreover, in light of these facts, almost no competitive journal would send out a paper for a third report if getting a 'reject' and a 'major R&R' recommendation from the first two referees. Prima facie, it would be weird if they did: it would suggest that the AE had an agenda in getting the paper published.

So in the case I mentioned in my previous post, I started with one referee in the paper's main area of specialisation; that referee returned a positive report. At that point, I normally read the paper carefully myself. (When I first receive a paper, I just skim it to come up with a list of prima facie appropriate referees.) In this case, at this point I realised that I'd need a cross-over referee in the adjoining area. So the second referee was the cross-over referee; and when that report was positive, I sought a third referee in the main area of specialisation. I was fortunate to be able to cover all these bases in the case.

You also asked whether an editor can reject a paper "even when it gets positive reviews". Yes, editors can and should exercise their own judgment, too. Usually this will be a matter of adjudicating what is in the actual reports, but conceivably one's own disciplinary expertise can play a role. It depends on the case. Moreover, authors should understand that, in most portals, reviewers can separate comments for the editorial team from comments that are intended for the author. A reviewer might find the paper promising on some level and want to be encouraging to the author, but also indicate to the editorial team that the piece has a long way to go, or may not be a great fit for the journal. Finally, as I am sure you are aware, many journals are incredibly competitive -- i.e. accepting fewer than 5% of submissions. These are crushing odds. A paper may come in that is genuinely a good paper, gets good reports, but is just not sufficiently groundbreaking perhaps, or perhaps not of sufficiently wide-ranging interest for a general philosophical readership, and gets rejected. If the AE is handling a paper that is not directly in his/her line of research expertise, these factors may not become evident until there are some referee reports in hand.

Finally, if your paper gets rejected, it really doesn't mean that it is a bad paper, or in some way unworthy. There is a skill in learning what one can even from uncharitable criticism; and there is a skill in learning what to act on, and not act on, in referee reports (no matter how charitable or uncharitable the report may be). Sometimes the most appropriate course of action is just to send it right out to another journal; sometimes it is not.