By Trevor Hedberg
If you stay in the profession long enough, eventually you submit your work to journals and receive comments on your work from reviewers. Unfortunately, reviewer comments are not only often negative but also unhelpful. In my own experience, when I get rejections with comments, I am usually only able to use about half of those remarks to improve the paper. In a few of these cases, I haven’t been able to use any of the reviewer comments for that purpose.
In a previous post on the topic of peer review, Jason Stanley mentioned some advice offered by Robert Nozick: simply send the paper along to the next journal, and don’t worry about the comments. There are circumstances where that strategy is appealing, but in the cases where a reviewer genuinely identifies a problem with the paper, I usually try to solve the problem before sending it somewhere else. (I understand the motivation to send as much material to as many journals as possible in quick succession, highlighted well in this post, but if I genuinely believe that the revision will improve the paper and think the revision can be done without needing to write a new paper altogether, I can't bring myself to ignore the criticism without feeling intellectually irresponsible.) In part, this is why unhelpful reviewer comments can be so frustrating.
By “unhelpful” reviewer comments, I am not referring to inflammatory or offensive comments. While those definitely aren’t helpful, I think of them as a separate category altogether. Here are a few examples of comments that I’ve encountered in the last few years (most more than once) that I considered unhelpful:
- The reviewer claimed that my paper contains no clear expression of its thesis statement. The thesis statement was in fact clearly stated near the end of the first paragraph on page 2 of the manuscript in the last few sentences of the introduction. In my experience, the default position of the thesis statement is the end of the introduction, so this criticism was baffling.
- The reviewer claimed that there was an obvious response to argument A and that argument A should thus be rejected. While I agreed with the reviewer, my paper acknowledged that fact and rejected argument A in the second half of the relevant section. The section was actually a defense of argument B, which builds on the rejection of argument A. It’s a safe bet that the reviewer did not read this section of the paper in its entirety.
- The reviewer claimed that I had misinterpreted the views of Philosopher X. I had sent Philosopher X a copy of my manuscript in advance of submitting it to the journal, and he acknowledged that my presentation of his views was accurate. So either the reviewer knew Philosopher X’s work better than Philosopher X himself or this criticism was misguided. I suspect the latter.
- The reviewer’s comments were too brief to say anything meaningful. In a few cases, I have received sets of comments that were so short as to barely provide any justification for the judgment that was made regarding the paper’s quality. In the most extreme case, a paper under review for 9 months received 2 sets of comments that were each 5 lines in length. Since neither reviewer had provided more than 4 sentences of comments, what was provided was too vague to be useful.
- The reviewer claimed that significant sections of the manuscript needed to be “more thorough” in their discussion. While I agreed with the reviewer to some extent, the paper was 7850 words, and the journal had a strict 8000-word limit. The reviewer offered no suggestions for what material (if any) should be cut to facilitate what would have been a massive increase in the paper’s length.
- The reviewers disagree. In one case, I had reviewers reach opposing verdicts about an entire manuscript: one pushing a minor revisions verdict and the other pushing a rejection verdict. (The editor ultimately adjudicated it by siding with the rejection verdict.) In other cases, I have had reviewers disagree about the merits of particular sections -- for instance, when one reviewer thinks the argument in section 3 is strong and the other thinks it is weak and should be drastically altered. It’s tough to improve a paper under these circumstances, since it can be difficult to determine which reviewer has the more plausible appraisal of your paper.
What’s obvious from these experiences (and the many others like them) is that the peer review process often fails to provide helpful feedback on submitted manuscripts. What is perhaps most frustrating is that reviewers can usually avoid leaving unhelpful comments without too much difficulty. (The lone exception is #6: it’s inevitable that reviewers will sometimes disagree about the merits of a particular argument, and I don’t see any clear solution to that problem.) Experiences #1 and #2 could have been avoided if the reviewer had simply read the paper more carefully, #3 could have been avoided if the reviewer had checked the work of Philosopher X before claiming I had misinterpreted it, and #5 could have been remedied if the reviewer had acknowledged the word limit and said something constructive about what to do about it. So what does it take to provide helpful feedback on submitted papers? I think it requires at least the following:
- Carefully read the work that you’re peer reviewing, ideally more than once.
- If the author is engaging significantly with the work of a particular philosopher, investigate their citations to determine if their interpretation of that philosopher’s work is correct.
- Take note of the paper’s word count and the journal’s length preferences when suggesting revisions.
- Ensure that your comments are sufficient to both justify the decision you have made regarding the manuscript and provide concrete suggestions that will help the author improve the paper. (This is the rule that the reviewers in #4 clearly violated.)
It’s also worth acknowledging that philosophy has a problem with inflammatory comments, an issue we’ve discussed here in the past. So we can safely add a 5th rule to this list:
- Refrain from using derogatory, offensive, or otherwise hurtful language in your comments. Furthermore, remember to only evaluate the paper and not make judgments about the author’s character or intelligence.
Now look back at that list. Is there anything about it that’s unreasonable? I hope not: these rules are remarkably similar to the same rules for reading and writing philosophy that are commonly offered to undergraduate students, and they follow naturally from the aims of trying to discover the truth about philosophical issues in a cordial, charitable, and productive way. Surely we should be willing to follow the same broad standards that we demand of our students!
This list of rules is surely incomplete. (At a minimum, there’s a need for a rule specifying how long one can permissibly take to peer review a manuscript.) Nevertheless, I suspect the value of the peer review process would be greatly increased if everyone were able to meet just these minimal standards. Anyone have other basic rules that they think all peer reviewers should follow?