What do you look for as a reviewer? And what should reviewers look for?

There have been a lot of posts recently, both here and elsewhere, on the perils of peer-review -- particularly on irresponsible, unaccountable reviewers, as well as on whether contemporary philosophy (and so, reviewers) have fetishized rigor to an unhealthy (and comical) extent.

In light of these issues, I'd like to ask ye, my fellow Cocooners, the following two questions:

1. What do you look for as a journal reviewer? (Please answer only if you've been a journal reviewer!)
2. What do you think journal reviewers should look for, and why?

I suppose I'll kick things off by giving my short-answer to both questions. 

• What I look for as a reviewer, and why: It depends on context. If I am reviewing a primarily critical paper or "negative argument" within a paper (one that takes another piece of philosophy to task), I look for rigor.  I want the author to really make a convincing case that the piece they're criticizing is in error.  On the other hand, if I am considering a piece of positive philosophy, I aim to balance rigor with interest, and sometimes prioritize the latter over the former.  Because I recognize that interesting new contributions to philosophy can be, and usually are, chock full of holes and failed arguments (see e.g. Kant, Rawls, Etc.), what I really want to know is: has the author motivated an interesting new way of looking at the problem(s) at issue?  Or, more bluntly, I ask myself: is this a paper people are likely to talk (and write) about?  (Notice: a paper can be one of this sort even if its argument has a lot of questionable components).
• What I think reviewers should look for, and why: I've made no secret of my belief that many reviewers focus far too much on rigor and not enough on interest.  I've read a lot of rigorous papers in top journals lately that I just don't think are very interesting at all (no, I won't name them!). My experience has been that Mumford is right.  Journal reviewers are far, far too conservative. When we prioritize rigor over "interestingness" -- when we insist that a person defends each and every premise of their "to the hilt" -- we end up publishing papers that are so conservative that they're just not that interesting.  There's a reason why Groundwork for a Metaphysics of Morals, A Theory of Justice, Quine's "Two Dogmas", etc., are so widely discussed...and it is not rigor.  As far as I am concerned, RM Hare's hilariously aggressive review of A Theory of Justice is more or less right in its details. Rawls' book is full of problems, left-and-right. But so much the worse for Hare. Rawls' book was interesting, and generated an incredible amount of conversation. Rigor is important. Interesting is more important. Or so say I.

I'm curious to hear what all of you think (and do yourselves)!

Comments

[Moti Mizrahi]

I can see why we want papers that are interesting and/or important published. But I wonder how reviewers make judgments about importance and interestingness. On numerous occasions, I've had one reviewer judge that a paper of mine is interesting/important and another reviewer judge that the same paper is not interesting/important. Any thoughts?

[Marcus Arvan]

Moti: I can't say how other reviewers make such judgments. I suspect (from personal experience) that some reviewers judge the importance of a paper in terms of how much it interests *them*. For example, I actually had a reviewer leave their name in the "properties" section of their review, and guess what? They said my paper was fundamentally misguided because...you guessed it, it simply went against their own view on the topic (nope, they did not say anything about my actual argument for my view).

In any case, when I judge the "interestingness" of a paper, I try to remove myself from the equation and ask myself: is this the sort of paper I think *others* would find interesting and worth writing on? If so, I judge the paper to be interesting.

[Moti Mizrahi]

Thanks, Marcus.

I've had a similar experience. A reviewer recommended rejection of a paper of mine which argues that Big Name's argument for p is fallacious. The reviewer's reason: "I never found Big Name's argument all that convincing." (Needless to say, Big Name's argument for p appeared in all the top journals in the field.)

[Kyle Whyte]

Perhaps there is a problem that reviewers are so entangled in their own interests that it would actually be hard from them to think outside of them. They might genuinely put in a good faith effort to evaluate an article going against their views according to its contribution to advancing the discussion in that area. But wouldn't their prejudices about the topic even frame what they would feel were worthwhile debates to have? They might think, for example, that their work on a topic ended the importance of debate on some matter, etc.

[Tom]

When I complain in a review that the paper's main thesis isn't interesting/substantial, I write something like "Publish if I'm the only reviewer who has doubts". It's never much of a sign for anything if one single person finds something not so interesting. I hope editors know that...

However, I came to this thread for a different reason: Could someone explain to me what is expected from a reviewer who gets to see the paper only after the first revision? I got the original paper and the revised paper, but not the statements from the other two reviewers. I think that I should only consider "reject" and "publish as is" as recommendations since I don't want to start another round of revisions. But maybe that is unfair as well. Any suggestions?

[Kyle Whyte]

So they gave you the paper, told you it was round two, but did not show you the original reviews nor do you have access to them through the online system? If that's the case, then I think you have to go with the situation the editor set up: the editor wants the article to get a fresh review from someone who has no idea what the first round was like. So the editor expects the full range of decisions.

[Marcus Arvan]

Tom: I definitely don't think it's appropriate to only consider "accept" or "reject" in that situation. If the paper is potentially publishable with revisions, you should recommend revise-and-resubmit. Remember, a person who has already received one R-and-R has already had their paper at the journal for a while, and they have it in a favorable spot there (they've already gotten a reviewer or two to judge it's potentially publishable with revisions). As someone who had an R-and-R at a journal after 8 months, did everything the reviewer asked for, and then had the paper rejected outright (by a new reviewer?) on what I thought were poor grounds, I can say -- for myself at least -- that I would rather go through another round of revisions than get a reject. Anyway, why hold yourself to accept or reject if, in reality, you think the paper *could* be published with revisions? Seems like an arbitrary self-imposed limit to me.

[Tom]

Thanks for the replies! My idea to only consider accept or reject is driven by the thought that as a second-round reviewer one should be more lenient and not introduce completely new demands for changes. As far as I gather the first two reviewer wanted A, B, and C changed and the author made the changes. Now I come along and my first thought after reading the paper is to ask for changes D, E, and F all of which are unrelated to A, B, and C. This is bound to be unsatisfying. As the author I would think "it took them all these months to ask for changes A, B, and C and now they start just another round; if that reviewer’s opinion is so important why didn’t they ask him right away?" Hence, I thought as a third reviewer one should ask for revisions only if they are really, really needed and the paper would in its present form corrupt the youth. After all, the paper should be available to all the participants of the debate and not be buried in the review process. Hence, my original suggestion: As a second-round reviewer one should ask for revisions only if the objections are so severe that one would usually recommend outright rejection. Does this still sound unreasonable to you?