Our books






Become a Fan

« Summer sounds | Main | How do you know when a paper is ready? »

08/09/2021

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

PT

I'm going to have to respectfully disagree with Marcus on this one. There's nothing wrong with saying that a paper is quite bad, and in fact I think you're obligated to do it.

First off, referees don't owe any obligation to the person whose paper they are reporting on. Instead, the referees' obligations are towards the journal and towards the profession at large. (If you disagree with the last disjunct, consider that "X does not make a novel contribution to the literature" is a good reason to reject paper X.)

I think this means that if a paper genuinely is quite bad, it is the referees' obligation to communicate this to the editors (and, if the editors deem it necessary, they'll pass it on to the author). I can't imagine thinking you have ANY obligation to say anything nice or "laudatory" about the paper. Moreover, if we think that referees have obligations to the profession at large, then if the paper really is very bad, it should be one's obligation as a referee to communicate this to the author, too, since it'll bring it to their attention that their work needs to be improved.

Also, re: Marcus's "avoiding being Referee #2" comment: I agree that we should avoid being Referee #2, but the "Referee #2" stereotype is that of an irresponsible referee who does not actually read the paper and leaves unhelpful comments. It is perfectly possible to be a responsible referee, and to leave helpful comments, if the paper is genuinely quite bad and the only helpful comment is: "This paper is bad and it should not be published." (Think of it this way: if a student's paper warranted a grade of a D or an F, you wouldn't think that it would be *irresponsible* to give it a bad grade, no?)

Marcus Arvan

Hey PT: To be clear, I am not suggesting that referees should avoid an honest evaluation of the paper. I often write critical reviews (and always honest ones). The real point, or so I think, is how one does it: snidely or respectfully.

Also, there have indeed been times where I haven't been able to find anything good to say about a particular paper. But, I think, this is pretty rare. For the most part, we are evaluating papers by professionals--people with many years (if not decades) of high-level training. If one regularly finds oneself having nothing good to say about papers as a reviewer, then I think that probably says more about the reviewer than the papers they are reviewing.

That being said, we evidently do disagree about something fairly fundamental here. You write, "referees don't owe any obligation to the person whose paper they are reporting on." I respectfully disagree. We owe everyone with whom we interact with decency and respect (the sole exception perhaps being when people behave so odiously that their behavior warrants the contrary). Not that we (or I) always live up to this standard--but, for all that, I think it is one worth striving to meet in all areas of human life.

And indeed, to address your student paper case, sometimes I do give D's or F's. That being said, as a rule, I have learned it to be better to give students with constructive rather than meanspirited or snide feedback. The D or F (and dispassionate justifications given for it) should suffice. The same, I think, should go for reviewing.

anon

"There's nothing wrong with saying that a paper is quite bad, and in fact I think you're obligated to do it."

Maybe I'm taking you more literally than you want to be taken, but "This paper is quite bad because [...]." is something I'd never write in a referee report. I would say instead, "My recommendation is 'reject' because of these three problems with the paper: [...]."

Chivers Butler

It might be helpful to distinguish two questions:

1) What may/should be said in an author-facing report?

2) What may/should be said in a report for the editor only?

My sense is that one can and should be quite blunt in the second case.

question for op

Shouldn't we be asking the OP about the respects in which the paper is bad? Is it poorly written? Rushed? Leaves very obvious counter-examples untouched? Doesn't cite those who they are academically and intellectually obligated to cite?

Or, do they mean by "bad": "I don't agree." The whole problem with bad reviews is that sometimes, that is what is going on. So what makes a paper this bad?

PT

First commenter here, replying to Marcus and others.

@Marcus: Obviously I agree that we owe basic norms of courtesy and respect. Likewise, I never said that one should give their students meanspirited or snide feedback. I don't think what I said could be reasonably construed as saying that referees or teachers can just be jerks simpliciter. I was just using a quantifier restriction.

All I meant is that *in their capacity as a referee*, a referee is under no obligation to make the author of a paper feel good about their paper, or to write a report that avoids being direct and honest about the (perceived) merits or faults of the paper. For instance, I was disagreeing with the suggestion that referees should "try to say a few laudatory things" about papers that are very bad.

Additionally, I wasn't suggesting that the OP (or myself, or anyone else) rarely finds themselves having nice things to say about a paper. But the OP specifically asked about a paper that is "very bad". I take this to mean something like the paper itself is just very unclearly written, or makes egregious mistakes concerning the established literature, or makes flat out factually incorrect claims. I think when these are the circumstances, it's better to be honest, and to say, e.g., "Look, this paper is just very unclearly written and the quality of writing in it just does not meet professional standards." I don't think that's meanspirited or snide, and in fact it seems to me to be well within what is expected of a responsible referee. On the rare occasion when I am asked to referee a paper that turns out to be very unclearly written, or which contains factual errors, or egregious misunderstandings of the literature, that is in fact what I write in my reports. I think that's warranted.

@Anon: Yes, that's exactly right. On the occasion when one has to referee a paper that is just really, painfully unclearly written, or which contains really bad misunderstandings of what's going on in some literature, I think it's quite appropriate to basically say: This paper should be rejected bc its writing is not up to professional standards / ...it badly misunderstands what is going on in the literature.

Marcus Arvan

Hey PT: I wasn't suggesting in the OP that we try to make authors feel good. The tips that I was offering are intended merely as rules of thumb to combat the (very real and pernicious) tendency of (some) reviewers to go overboard to (unnecessarily) make authors feel bad. I still think they are good rules of thumb, including the thing about trying to say something positive.

Why? Well, see the study that I link to in the OP on how common inflammatory comments are in philosophy reviewer reports. Or reflect on some of the reviewer reports you've received (since virtually everyone I know mentions receiving reviewer reports that seem unfair and needlessly inflammatory to them!). Finally, if you haven't read it, I really recommend reading the piece that I linked to in the OP on rejected Nobel-prize winning economics papers. It's a proverbial eye opener.

When academic journal reviewers have a documented tendency to reject future *Nobel-prize* winning papers as "trivial" or (I kid you not, this is a real one) "This amounts to nothing and should be refused", there's a problem, right? The problem isn't that referees are too honest. It's that sometimes they needlessly (and erroneously) go out of their way to make authors feel terrible about themselves. So, or so I'm inclined to think, we should adopt strategies to combat these tendencies.

Sometimes, there really is nothing positive to say about a paper. But, the vast majority of the time, my sense is that there is *something* good about a paper if it pasts desk-review--so the tip there (to try to say something laudatory) is, I think, a good rule of thumb to try to bear in mind.

Anyway, that's what I took the OP to be asking for tips about when they mention "the research about the damaging effects overly that negative reviews can have on people - and I don't want to be that reviewer!" They were merely asking how to not be a jerk ("How does one best communicate a negative verdict without going over the line?"). So, if you agree that we shouldn't be jerks, we're on the same page--though here again, I'm with anon: it's all about how you say what you say.

Anon UK Grad

Op here, answering 'Questions for OP'.

I don't just mean that I disagree with the paper (though that also happens to be true) - and, for what it is worth, I have happily recommended acceptance for papers that I disagree with in the past. Refereeing, in my view, is not about enforcing my own views, but rather about ensuring suitable quality.

But, in any case, I don't want to say much about the specifics of the paper - I am hoping the discussion can be general enough to be helpful to others who find themselves in this situation as well.

Tim

OP wants advice regarding "how to write negative journal reviews without being too harsh in the tone." From one perspective, this is impossible. A report where you reject a paper is *always* harsh in tone--its a rejection! Beyond that, I think the recommendation of neutral/non-inflammatory language is best. Its the most effective way of minimizing rudeness.

I'd also second the recommendation to divide out comments for author and comments for editor. I once was asked to reviewer a really bad paper. In my report, I wrote about 1000 words on the first three pages, detailing various problems in scholarship. But to the editor I said that I only stopped after three pages because I reached a 1000 words. I could have kept going for much longer on the rest of the paper. The author didn't need to hear that; it may have just upset them. But I thought it was important to communicate to the editor that my low opinion of the paper outstripped just what I had written. (I have even heard of some cases where Reviewers simply provide no comments for the author and tell the editor they don't even think the paper should have been circulated!)

zebra

I think the practice of writing separate comments to the editor is what leads to both abuse, and those getting their papers rejected thinking the system is not opaque. If you cannot say it in a way that the author of the (bad) paper should not see it, then do not say it. But really, do let the author know their paper does not meet professional standards.

Prof L

I would add: don't hypothesize about the author's career stage, gender, who has helped them with the paper or not, their nationality, their dissertation advisor, their level of education, their first or second language, their preparation or background, what they do not know, and so on. You're probably wrong, it's totally irrelevant, and can also get very nasty and personal.

I would say it goes without saying, but, well ...

Relatedly, "The author clearly does not understand X" or "The author knows nothing about Y" are speculative. Such things can be rephrased to be about what the paper demonstrates, or into a statement about why something that was not included should be included.

Humility

In dialing back earlier comments that appeared to defend excessively harsh criticism as sometimes deserved, PT says:

"I think it's quite appropriate to basically say: This paper should be rejected bc its writing is not up to professional standards / ...it badly misunderstands what is going on in the literature."

These kinds of comments are more boarderline, but for what its worth, I don't think you should be writing such comments either in a referee report.

Saying to a fellow professional "your writing isn't up to professional standards" is unnecessarily harsh. Better to say "the writing in this paper is unclear in these specific ways ... these shortcomings are significant enough for me to find this paper unsuitable for publication. [Aside: I've reviewed 30+ papers and although several have been poorly written, I've never had one written so badly that the writing alone was a reason for rejecting (as opposed to giving them a chance to correct it in an R&R). If you find yourself regularly rejecting papers because they are "poorly written" then the problem may be with your standards and mindset rather than with the papers themselves.]

Saying that someone "badly" misunderstands the relevant literature is also unnecessarily harsh. You should instead say "The paper misunderstands the relevant literature in these specific ways ... These misunderstandings are significant ones that undermine the papers relevance in this way ..."

Two general points to draw from this:

(1) Starting with broad sweeping statements about why the paper is bad or poor in certain basic ways often gives an overly negative tone. Better to start with milder statements "there is a problem with x" then give specifics about exactly what the problem is, and, only after these specifics have been given, explain why they are major flaws that mean the paper cannot be accepted.

(2) Understand that tone and color matter just as much as literal meaning. The sentence "this paper badly misunderstands x" may have roughly the same meaning as "There is a significant misunderstanding in this papers account of x" and "this paper misunderstands x in fundamental ways" but its tone is much harsher. On some psychological level that goes beyond semantics and pragmatics it says to the author "you have done a bad job, you have done badly here".

Jonathan Ichikawa

I just want to reiterate the key thing that Chivers Butler said, which is pretty central: you can and should use a very different tone to the editor than you do to the author.

You can say "here is why the author's project is interesting, and here are my reservations about accepting it for publication" to the author, and say "to be frank, this paper is just quite bad, and I don't see any reasonable prospect for improving it to publishable standards" to the editor.

Michel

The first paper I was ever asked to review was very bad. I wasn't entirely certain it wasn't a hoax paper. I said as much to the editor, but otherwise tried to engage with it genuinely, and left it serious, neutrally-expressed comments.

The only other time it's happened for me was refereeing a book. It was an absolutely dreadful piece of scholarship (to the extent it even was 'scholarship'). I did the same then as I did that first time.

That's twice in about ~30 sets of referee reports, so it's been quite rare (and I've never refereed a conference paper as bad as those, and I've refereed a lot more conference papers than articles and books!).

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)

Job-market reporting thread

Current Job-Market Discussion Thread

Philosophers in Industry Directory

Cocoon Job-Market Mentoring Program

Categories