In our newest "how can we help you?" thread, a reader writes:
I was asked by a tippy-top journal about a year ago to review a paper on a topic that is squarely in my wheelhouse. I'm one of a pretty small number of people extremely well suited (research-area wise, at least) to review the paper. Because it's a tippy-top journal with an extremely low accept rate, I spent a LOT of time on the review. I try to be very conscientious about reviews generally, but this was especially careful. My review was about 10pp. Note: I'm tenured at a research institution. I returned the review promptly.
There were some very big mistakes--it was a good idea, but had some very serious problems. I offered a lot of comments about how to avoid the problems, some key pieces to cite, and suggested that with those improvements it was a publishable piece (if not at that tippiest toppiest journal). THe journal shared the EIC decision and the other reviewer's comments. The other reviewer's concerns overlapped with mine.
Now I have received a request to review the paper again from another (excellent, if not quite as rarefied) journal. The paper was attached in the request, and it has not changed. Like, none of the very important problems I pointed out have been addressed at all. None of the other reviewer's excellent comments were addressed. My guess is that the paper went straight back out to at least one (possibly two) other excellent journals without addressing the comments (on which I, and at least one other scholar--maybe one of you!--spent a great deal of time), was rejected again, and is now coming back to me.
So, I'm a bit irked by that. This person is clogging up the review process. I know that lots of advice around here is to turn around articles quickly, not spend too much time meeting reviewers' idiosynchratic criticisms, etc. But this strikes me as a case of serious free-loading, and it makes me reluctant to review papers (or to review them conscientiously).
I guess this post is mostly a rant--sorry about that. But I do need advice. How do I respond?
a. I could simply decline to review with no reason given. No doubt I'm biased in that I thought my own comments were good.
b. I could decline but warn the EIC that this appears to be a person just shotgunning their paper out there, and asking others to review is a burden that won't result in a modified paper.
c. I could neither accept nor decline and share my prior review with the EIC.
d. else?
There were already a few replies to this comment in the read, but I'd like to hear what more people think, as I suspect this happens a lot. Ref replied:
Just tell the Editor that will not review the paper because you reviewed the paper for another journal, and it has not changed, despite the many comments you provided.
anon then added:
I understand your frustration. But I've been on the other side of a similar situation, so here's my own take on what went down in my case:
(1) I got rejected from Journal X, with a lot of comments and the assertion that even if I revised the paper, it would be publishable but not in as elite a place as Journal X.
(2) I read the comments and made the judgment call that they weren't necessary revisions.
(3) I submitted the paper to equally prestigious Journal Y.
(4) I received an R&R from Journal Y, with very, very different comments. (i.e., attending to the previous comments would have done nothing to help me with the objections I received from Journal Y.)
(5) I made those revisions and published the paper.
Obviously, what matters is how truly necessary the comments are - maybe you're right, and the author really should have revised their paper. But this is something that epistemic peers disagree about quite a lot.
As for what you should do in your case: if you're now willing to give this author an R&R, maybe you can work things out with them in the back and forth where you get to see the revision letter, i.e., the author's reasoning concerning your comments.
Finally, Amanda wrote:
If I have a situation like that I do not accept or decline, but I write the editor and say, "I have already reviewed this paper for another journal." And then I add one comment about what I thought about the paper. If I thought it was a bad paper and it hadn't changed, I would say, "My review was not favorable."
I kind of think it's fair to give the paper to a new reviewer, given all the disagreements in philosophy among equally smart and equally qualified persons. I also wouldn't hold it too much against the author for not changing anything. Maybe they were told by others they trust that the paper is fine as it is, maybe they are coming up for tenure and didn't have time to make the changes and decided to hedge their bets. I just don't think it's the referees job to judge the decision not to make changes. It is the referees job to judge the paper.
I'm on the same page with these readers. Here's one thing that I have heard of happening before (indeed, I suspect it has happened to me before). You draft up a paper, let's say it's 8,000 words - the maximum word count for one tippy-top journal, Mind (see here). The paper got a good response at conferences, but the first journal you send it to, reviewers find 'problems' and so you revise the paper to 10K words to address those concerns. That lands you at the word-count limit for a bunch of other journals. So, you send it to some of those journals, and the reviewers find more problems. So, you revise and now the paper is 12K words. You see where this is going, right? The more reviewers who read a given paper, the more problems they are likely to find--and you'll keep revising the paper upward, have fewer journals you can send the paper to...and the new reviewers at new journals may be likely to find new problems they think you need to address!
I recently heard someone (I think it was Mike Huemer) suggest this is less a problem with authors than it is with how some (many?) reviewers regard their task as reviewers. Many reviewers seem to think that if a paper has 'problems', then they should ask for revisions or reject the paper outright. But, Huemer suggested (and I agree), this isn't how reviewers should regard their job. Every paper, even many of the most influential ones in the discipline, have problems. The question shouldn't be whether a paper has problems, but whether it is worth thinking about and exposing a journal's readers to. Now, of course, if as a reviewer you find a paper totally uninteresting or its argument clearly bad, then sure, recommend rejection. But I don't think we should be rejecting every paper that has problems, or require every problem to be addressed in revisions. Problems are not in and of themselves reasons to reject papers. They may very well be interesting problems for the subsequent literature to find and address (e.g. in response papers, etc.).
More generally, I am inclined to think that we, as reviewers, should err on the side of epistemic humility. There are just too many documented cases of reviewers being "sure" a given paper has serious problems...and those very papers either literally going on to win the Nobel Prize (!) or otherwise going on to become classics in their field. If you think a paper is bad, feel free to say so in a review. But, if you get asked to review the paper again, do everyone a favor and just decline: give another reviewer an opportunity to review the paper. If the paper is as bad as you think it is, then another reviewer will almost certainly find the same issues you found and reject the paper again. I understand cases like this may seem like they 'clog the system', but honestly I think this is just the price we pay for a bad publishing system.
Anyway, these are just my thoughts. What are yours?
Recent Comments