Unhelpful Reviewer Comments

By Trevor Hedberg

If you stay in the profession long enough, eventually you submit your work to journals and receive comments on your work from reviewers. Unfortunately, reviewer comments are not only often negative but also unhelpful. In my own experience, when I get rejections with comments, I am usually only able to use about half of those remarks to improve the paper. In a few of these cases, I haven’t been able to use any of the reviewer comments for that purpose.

In a previous post on the topic of peer review, Jason Stanley mentioned some advice offered by Robert Nozick: simply send the paper along to the next journal, and don’t worry about the comments. There are circumstances where that strategy is appealing, but in the cases where a reviewer genuinely identifies a problem with the paper, I usually try to solve the problem before sending it somewhere else. (I understand the motivation to send as much material to as many journals as possible in quick succession, highlighted well in this post, but if I genuinely believe that the revision will improve the paper and think the revision can be done without needing to write a new paper altogether, I can't bring myself to ignore the criticism without feeling intellectually irresponsible.) In part, this is why unhelpful reviewer comments can be so frustrating.

By “unhelpful” reviewer comments, I am not referring to inflammatory or offensive comments. While those definitely aren’t helpful, I think of them as a separate category altogether. Here are a few examples of comments that I’ve encountered in the last few years (most more than once) that I considered unhelpful:

1. The reviewer claimed that my paper contains no clear expression of its thesis statement. The thesis statement was in fact clearly stated near the end of the first paragraph on page 2 of the manuscript in the last few sentences of the introduction. In my experience, the default position of the thesis statement is the end of the introduction, so this criticism was baffling.
2. The reviewer claimed that there was an obvious response to argument A and that argument A should thus be rejected. While I agreed with the reviewer, my paper acknowledged that fact and rejected argument A in the second half of the relevant section. The section was actually a defense of argument B, which builds on the rejection of argument A. It’s a safe bet that the reviewer did not read this section of the paper in its entirety.
3. The reviewer claimed that I had misinterpreted the views of Philosopher X. I had sent Philosopher X a copy of my manuscript in advance of submitting it to the journal, and he acknowledged that my presentation of his views was accurate. So either the reviewer knew Philosopher X’s work better than Philosopher X himself or this criticism was misguided. I suspect the latter.
4. The reviewer’s comments were too brief to say anything meaningful. In a few cases, I have received sets of comments that were so short as to barely provide any justification for the judgment that was made regarding the paper’s quality. In the most extreme case, a paper under review for 9 months received 2 sets of comments that were each 5 lines in length. Since neither reviewer had provided more than 4 sentences of comments, what was provided was too vague to be useful.
5. The reviewer claimed that significant sections of the manuscript needed to be “more thorough” in their discussion. While I agreed with the reviewer to some extent, the paper was 7850 words, and the journal had a strict 8000-word limit. The reviewer offered no suggestions for what material (if any) should be cut to facilitate what would have been a massive increase in the paper’s length.
6. The reviewers disagree. In one case, I had reviewers reach opposing verdicts about an entire manuscript: one pushing a minor revisions verdict and the other pushing a rejection verdict. (The editor ultimately adjudicated it by siding with the rejection verdict.) In other cases, I have had reviewers disagree about the merits of particular sections -- for instance, when one reviewer thinks the argument in section 3 is strong and the other thinks it is weak and should be drastically altered. It’s tough to improve a paper under these circumstances, since it can be difficult to determine which reviewer has the more plausible appraisal of your paper.

What’s obvious from these experiences (and the many others like them) is that the peer review process often fails to provide helpful feedback on submitted manuscripts. What is perhaps most frustrating is that reviewers can usually avoid leaving unhelpful comments without too much difficulty. (The lone exception is #6: it’s inevitable that reviewers will sometimes disagree about the merits of a particular argument, and I don’t see any clear solution to that problem.) Experiences #1 and #2 could have been avoided if the reviewer had simply read the paper more carefully, #3 could have been avoided if the reviewer had checked the work of Philosopher X before claiming I had misinterpreted it, and #5 could have been remedied if the reviewer had acknowledged the word limit and said something constructive about what to do about it. So what does it take to provide helpful feedback on submitted papers? I think it requires at least the following:

1. Carefully read the work that you’re peer reviewing, ideally more than once.
2. If the author is engaging significantly with the work of a particular philosopher, investigate their citations to determine if their interpretation of that philosopher’s work is correct.
3. Take note of the paper’s word count and the journal’s length preferences when suggesting revisions.
4. Ensure that your comments are sufficient to both justify the decision you have made regarding the manuscript and provide concrete suggestions that will help the author improve the paper. (This is the rule that the reviewers in #4 clearly violated.)

It’s also worth acknowledging that philosophy has a problem with inflammatory comments, an issue we’ve discussed here in the past. So we can safely add a 5th rule to this list:

• Refrain from using derogatory, offensive, or otherwise hurtful language in your comments. Furthermore, remember to only evaluate the paper and not make judgments about the author’s character or intelligence.

Now look back at that list. Is there anything about it that’s unreasonable? I hope not: these rules are remarkably similar to the same rules for reading and writing philosophy that are commonly offered to undergraduate students, and they follow naturally from the aims of trying to discover the truth about philosophical issues in a cordial, charitable, and productive way. Surely we should be willing to follow the same broad standards that we demand of our students!

This list of rules is surely incomplete. (At a minimum, there’s a need for a rule specifying how long one can permissibly take to peer review a manuscript.) Nevertheless, I suspect the value of the peer review process would be greatly increased if everyone were able to meet just these minimal standards. Anyone have other basic rules that they think all peer reviewers should follow?

Comments

[Eugene]

Nice post, Marcus.

I agree that 4-6 are unhelpful comments to receive, but 1-3 seem fine to me.

"1. The reviewer claimed that my paper contains no clear expression of its thesis statement. The thesis statement was in fact clearly stated near the end of the first paragraph on page 2 of the manuscript in the last few sentences of the introduction. In my experience, the default position of the thesis statement is the end of the introduction, so this criticism was baffling."

This is baffling? The lesson here is that in addition to including a thesis at the end of the introduction, include it again (and perhaps again and again), elsewhere.

"2. The reviewer claimed that there was an obvious response to argument A and that argument A should thus be rejected. While I agreed with the reviewer, my paper acknowledged that fact and rejected argument A in the second half of the relevant section. The section was actually a defense of argument B, which builds on the rejection of argument A. It’s a safe bet that the reviewer did not read this section of the paper in its entirety."

The lesson I would take from such a comment is that it would be useful to readers to remind them, perhaps in a footnote, why the "obvious response" is a bad or irrelevant one.

"3. The reviewer claimed that I had misinterpreted the views of Philosopher X. I had sent Philosopher X a copy of my manuscript in advance of submitting it to the journal, and he acknowledged that my presentation of his views was accurate. So either the reviewer knew Philosopher X’s work better than Philosopher X himself or this criticism was misguided. I suspect the latter."

Sure, but Philosopher X might not themselves present their views in a way where people can easily pin them down, so in a way it doesn't matter whether X thinks your interpretation is accurate. Lesson? Include a footnote to the effect that "it might be tempting to interpret X this way, but...".

All of these lessons seem to me to fall out of a moral general injunction to minimize cognitive demand on one's readers. It's often difficult to remember complex philosophical claims, and readers shouldn't have to scroll back to page two to recall them.

[Eugene]

Sorry, I got ahead of myself and meant to say nice post Trevor, not Marcus.

[Marcus Arvan]

Hi Trevor: Thanks for the great post!

I think, as Eugene's comment suggests, your post is useful for two things:

(1) Drawing attention to some common reviewer errors.

(2) Understanding how to "play the publishing game" better.

My experience as an early-career philosopher learning the publishing ropes is that it is *really* important to link these things together. Allow me to explain.

Early in my career, when I was having trouble publishing, I too was similarly perturbed by what I took to be unhelpful/mistaken reviewer comments. However, I also learned that there were certain *patterns* to how reviewers read papers, and that if one wants to publish successfully, one must anticipate and head off reviewer misreadings (the point of Eugene's comment!).

One general thing I've learned--which seems supported by the unhelpful tendencies you report experiencing--is that one must be *really* clear to reviewers, so as to head off misunderstanding. Among other things, one needs to:

(A) Make one's thesis as clear as possible.
(B) Say very clearly, and up front, how the thesis adds something of value to the literature.
(C) Head off potential reviewer concerns very early in the paper, before they have any chance to arise/bother the reviewer.

These are just a few things, but again, I found they were very important lessons to learn from reviewer responses. For instance, one of my now-published papers bounced around journals for years, despite the fact that I thought it dealt well with most of the objections reviewers raised. Then I had one reviewer tell me, "You dealt with objection X really well, but it came too late in the paper, and I had been worrying about the objection the whole time I was reading."

In short, I suspect that the more experience you have with "unhelpful reviewers", the more you'll learn how to anticipate their "unhelpful" tendencies, and actually, as an author, help *them* avoid their unhelpful tendencies. This, at any rate, has been my experience!

[Trevor Hedberg]

Hi, Eugene. Thanks for your comments. You may be right about #3, at least in some cases. That particular instance is too far in the past for me to know whether that's a reasonable interpretation of what happened. I'm more skeptical about your claims about #1 and #2, though.

Regarding #1, the thesis was stated at other places in the paper, and since the paper was accepted (pending minor revisions) at a journal of comparable quality shortly thereafter (with virtually no changes made to the manuscript), I'm inclined to think that the reviewer wasn't reading the paper closely.

As for #2, the problem is that the reviewer claimed that I was endorsing argument A when a central point of the section was to reject argument A. Given that the relevant section was fairly short (about 800 words) and that this point was not made in a convoluted or mysterious way, I couldn't see how a reviewer could come to that judgment without either not reading the entire section or not reading the section carefully.

My broader concern is that we teach our undergraduates to interpret philosophers charitably and to try to understand their arguments as well as we can given what the authors have said but appear to frequently fail to meet that standard in peer reviews of other philosophers' work. While I agree with the general injunction that you mention (i.e., to reduce the cognitive demands on the reader), I don't think all cases akin to what I've described are attributable to a failure on the author's part to meet that standard.

[Trevor Hedberg]

Hi, Marcus. I agree with your (A)-(C) list and have heard similar advice from other sources. I imagine that one does get better at preemptively addressing reviewer concerns as one gains experience. Simultaneously, even the most productive and successful philosophers I know encounter unhelpful reviewer comments with some regularity. In those cases, they know the ins-and-outs of publishing as well as most anyone does, so it's definitely reasonable for us to strive to do a little better as peer reviewers than we are in the status quo.

[Marcus Arvan]

Hi Trevor: I totally agree! Like most people, I still get reviewer comments that drive me batty--and we very much should encourage helpfulness among reviewers. I think your post is very...helpful (pun intended) in this regard. I merely meant to point out that, at the same time, paying attention to common forms of reviewer feedback, even the unhelpful type, can help one improve as an author, helping one recognize that there are certain types of errors that readers are apt to make, errors that one can develop authorial strategies to avoid!

[postdoc]

Part of the problem with the review process is that the hyper competitiveness of the discipline is forcing everyone, including MA students to submit papers to journals. There just are not enough quality philosophers to review all of this work. So, journals resort to lower and lower quality referees. I have gotten many reports from referees who were clearly incompetent. (I am not saying this out of spite. I have been successful at publishing in top journals.)

[Scott Forschler]

I have received a number of very helpful comments from reviewers over the years, some of which led to revision, others leading me to at least temporarily abandon the article. I have been also been astounded by the gross stupidity of some reviewers. Recently the chief editor at a major journal told me there had been too much published on the topic of my article recently, when my own literature review concluded that there was almost nothing published on that precise topic, ever. When I asked for an example, the editor gave one example--which was actually only an only marginally-related topic, and published 50 years ago! Off to the next journal, then. And several times I've gotten comments which reveal that the reviewer only skimmed the article and misunderstood it in elementary ways, sometimes even accusing me of making incorrect assumptions when the falsity of those assumptions was, rather, the precise basis of my criticism of other authors!

These experiences are apparently legion. I highly recommend Mark Twain's own amusing experiences on this issue, related in his delightful story here. http://www.twainquotes.com/Galaxy/187007a.html

[Scott Forschler]

How can I find journals who could benefit from additional book or article ms reviewers? I have more free time than most philosophers and my usual turn-around time for review requests is 1 week for books, 1 day for journal articles. I would love to do more--hook me up!