Shay Logan (postdoc, NC State) wrote in with the following interesting query:
We've all received horrible referee reports. I'm curious if any of us has ever *written* a horrible referee report and then realized it. That is, have you ever written a referee report, then (for example) on receiving the author's responses come to realize that you'd misunderstood the paper in some completely boneheaded way?
Bonus points if you can say what made you realize you were at fault.
I think it could be very instructive to hear readers' experiences. It might help us all better understand what being a "bad reviewer" amounts to! So, then, have you ever been a bad reviewer? How? Alternatively, if you haven't been a bad reviewer but have been on the receiving end of bad reviews, what exactly was bad about them? I suppose I'll begin...
It is hard to remember most of the reviews I've written. Fortunately, of the ones I do remember, I'm not sure that I've written any that grossly misunderstand the author's argument, etc. One problem here, of course, is that it can be hard to know if one has grossly misunderstood a paper (if one knew it and one wasn't being willfully obtuse, one would avoid it!). Still, I work very hard on the reviews I complete, trying to make sure I get things right. In any case, I do recall one review I wrote a number of years ago that stands out. It was a very bad review in the sense that I essentially argued that the author's argument was a non-starter, as well as inept at a scholarly level (ignoring vast swaths of literature). Because it's the only review like that I've ever written, and the content was pretty harsh, I went back several times to temper the review's language, so that at the very least I did what I could to not come across as needlessly mean (though I wouldn't be surprised, given the review's content, if the reviewer still interpreted the review as mean).
Fortunately, as I said above, that's the only review like that I've ever written. Generally speaking, I try very hard to write "balanced" reviews--ones that, first, lay out the essentials of the author's argument; second, say something complementary about the paper (as there is usually something worthwhile even in papers that are currently publishable); and then, finally, justify my recommendation in detail by engaging with the author's argument.
Much more often, I've been on the receiving end of awful reviews. Typically, the kinds of bad reviews I've received are the 4-5 sentence sort--the sort that, after I've waited anywhere from 2-12+ months, simply dismiss the paper as unpublishable without any detailed explanation why (often simply saying things like, "This paper is too ambitious, and does not succeed in fulfilling its ambitions"). Interestingly, albeit anecdotally, my sense is that I've gotten these types of reviews much more often when submitting to highly ranked journals. I've tended to notice that most of the "good reviews" I've gotten--ones that really engage my work in detail--are from reviewers at lower-ranked journals. Although this is anecdotal, I do have a sense as to why this might be. I have heard a few times that some journal editors simply "trust the judgment" of some reviews, particularly really established figures in the field--not expecting detailed reviews from them because they are so notable. Personally, whatever is the case, I think 4-5 sentence reviews are unconscionable and should in no case be accepted by editors. If a paper is bad, a reviewer should be able to provide some actual detail as to why--not just be able to say, "Trust me, it's bad." There are many reasons for this. As established as a given person may or may not be, they are also human--with human fallibility, biases, etc.
The other kinds of bad reviews I've received are the willfully meanspirited ones--ones which go out of their way to insult the author or the work in question. And I know I'm not alone in these either. I know at least one leading figure in the field who told me a referee told him a recent paper of his was one the reviewer would expect from a grad student. Finally, the most amusing bad review I ever received was for my 2013 paper, 'A New Theory of Free Will.' When I first submitted it to another journal, it was about 21 pages long. I got a revise-and-resubmit, and the reviewer's comments were so helpful that I rewrote it so that it was 78 pages long. I figured since the journal had no official word-limit, I might be able to get away with it. Um, no. The reviewer was none too pleased, writing a single sentence to the effect of, "I'm not going to read a 78 page paper." Unfortunately, the paper was rejected--but on the other hand, I got a good laugh out of it and published it elsewhere! :)
Anyway, what are your experiences either writing or receiving bad reviews? What constitutes one?
Some of my bad reviews I've received I would have been happy to have gotten 4-5 sentences. Instead I got 1-2 very mean-spirited sentences!
I do think there was a couple of reviews I have given that I did not make enough of an effort to be positive and helpful. What a lot of reviewers should think about is often just stating the facts (which, naturally, are often critical) can come across as very rude. I think many reviewers don't put enough thought into things like that. Writing a good and helpful review takes effort, but I think we have moral obligation to do so.
Marcus I have heard you tell the story about your 78-page paper a number of times, and to be honest I would have said the same thing as the reviewer. (Even though, I like to consider myself nicer and more generous than the average reviewer.)
What doesn't seem to cross your mind is that long papers can be bad not for any scholarly or philosophical reasons, but for human and practical reasons. Many people simply do not have the time, patience, or motivation to read a 78 page paper, no matter how wonderful it might be. I think since you are the sort of person who works like a maniac (given your descriptions of how much work you put into your students) perhaps you don't take into account the average philosopher's perspective. I admire how much you work, but I don't think it's morally necessary to do what you do. And I do think as philosophers we should try to consider how best to have a conversation with other philosophers. Other philosophers are persons with lives and work and personal limitations. That is why I try to write short papers: I want to have actual conversations with people, and I know they won't read my work if it is too long. Anyway I have thought this for a while so I thought I would share my perspective.
Posted by: Amanda | 06/07/2017 at 01:11 PM
I've never received a bad review - at least not in the sense that the comments were mean and hostile. But I have received very short reports - just a few sentences - but I don't consider those bad. Sometimes they say "the paper is mostly secondary, responding to X or and doesn't develop much of their own positive view - it merely criticizes X." or "it seems pretty similar to Y's view" and I think "that sounds right". I don't think it is the referees job to necessarily provide detailed comments. All I want is some reason why it wasn't published - "Not original enough" or "not in the scope of this journal" or whatever, is good enough for me. I send it somewhere else, or revise it.
I have received reports where I thought the author didn't understand my point -but I've always thought: I could've been clearer, and so I revise. I've been lucky never to have received a truly incompetent report (e.g., where the referee didn't appear to read the paper). Maybe it helps that I don't work in really contentious areas like ethics or political philosophy.
Have I written any bad reviews? I'm careful not to be mean - I do sometimes write short reports of only a few sentences, so I guess by Marcus' account those count as bad.
One manuscript I was asked to referee cited only two papers. There was a sizable literature on the topic (30-40 papers at least). So I wrote a very short report, along the lines of "This paper only cites 2 references, and the point the author makes seems to be made elsewhere, in X, which the author doesn't cite." Am I really required to write a detailed reply in such cases? Of course, I suppose I could've written a long reply about whether what the author said is exactly the same as what X says - but when the author hasn't done their homework - they don't tell me how/why their contribution is original by situating it in the literature - is it really the referee's job to do that? (I was tempted to add something snarky like "was this written by a student?" but it was more likely written by someone outside the area or outside philosophy or academia altogether). I resist all such temptations to be insulting once I submit the report.
I also wrote a slightly grumpy report when asked to referee a 20,000 word paper. I provided 5 or 6 pages of comments, but the main problem (unlike Marcus' long paper) is that this one rambled and could easily have been half the length.
Now that I get asked to referee a lot more (compared to early in my career), I tend to write shorter reports, but I'm also more generous with what counts as publication worthy. I probably recommend half of the papers I referee for publication. I don't reject it because I think it has a problem or flaw; instead, I ask myself: is it comparable to the average paper published in this journal? is it original? (I tend to think many journal articles have flaws, so...) If yes, I usually recommend it for publication and tell the editor that (along with whatever other comments I have).
But I have no idea whether I'm getting a random sample of papers (are many desk rejected?)
So I don't mind short reports. The only thing I mind is if the journal takes a long time to get back to me.
Posted by: Chris Stephens | 06/07/2017 at 02:48 PM
I suppose I should give my two cents since Marcus took the time to post my question (thanks, btw!).
I've definitely received bad reviews. I've also definitely mistaken ok reviews for bad reviews by misunderstanding the reviewer. The worst review I've ever gotten went something along the lines of "When I saw extremely common word W in the title of the paper, I assumed it meant X from obscure field Y unrelated to the declared subject of the paper. But if W means X then the paper doesn't make sense. In particular the definition of W given on page 2 is wrong, because it doesn't define W to mean X. And if W does mean X, as I assumed it did after reading only title, then saying W means something else is wrong. So the paper should be rejected."
Luckily in that case, the editor intervened. (She basically said something like "Erm... no," but in an official and intelligent-sounding way.)
I guess I asked the question because I assume that (a) I'm roughly an average reviewer and (b) the people who have reviewed my papers are (on average) average, so (c) I'm probably (proportionally) giving out as many awful reviews as I get. I'd like to not be doing that. But of course this requires knowing how to identify awfulness in the reviews I write, which seems hard...
Posted by: Shay Allen Logan | 06/07/2017 at 04:11 PM
I get asked to referee a lot (15-20 papers a year). I used to do about 15 a year, now, though, because of other work demands, I only referee about 8 or 10 papers per year. I decline whenever the paper is long. In fact, I even ask editors when they invite me to referee how long the paper is. Anything over 9000 words I generally say no. I once refereed a paper that was 37,000 words. I was so angry because so much of it was unnecessary, and this is almost half a short book. That is not what I agreed to when I said I would referee for a journal.
Also, I am now more choosy about the topic. I used to be willing to referee more broadly, but because of these other constraints, I now focus more narrowly on stuff related to topics in which I am well versed.
Posted by: Striped shirt | 06/08/2017 at 04:33 PM
Striped shirt: 37,000 words!? That's like 140 double-spaced pages. I mean, I know I sent out a 78-page paper myself (which I would review too, though I know it's pushing it)...but 140 pages? I'm surprised you didn't refuse. Surely the line has to be drawn somewhere. ;P
Posted by: Marcus Arvan | 06/08/2017 at 06:36 PM