A quandary for an evolving journal reviewer

In his post yesterday on some reasons he rejects papers, Trevor wrote:

When I first started peer reviewing a few years ago, I thought it would be very difficult to differentiate between papers that warranted R&R verdicts and those that should just be rejected. I wound up being wrong about this for the most part. (There has been only one paper I’ve reviewed that I think was genuinely on the borderline.) The reason I was wrong is that most papers I review feature glaring problems, some of which should be addressed well before the paper is submitted to a journal.

My experience as a reviewer is very different, and indeed, evolving--albeit in a way that has landed me in a bit of a quandary as a reviewer. Allow me to explain, and then ask for some advice!

In this illuminating (and caustic) commentary on publishing in philosophy, Mike Huemer argues:

When refereeing a paper, the question to ask is not, “Is this convincing?” The question is, “Does this add to the discussion? Would people like to talk about it?”

The more that I reflect on Huemer's reasoning--which includes these anecdotes about how some of his influential works were systematically rejected (a tale similar to Jason's Stanley's here), not to mention all of the classic and Nobel Prize-winning articles rejected in economics--the more it seems right to me. Just about every published paper in philosophy has serious problems, including the very best ones. For example, I'm teaching Frege, Russell, Davidson, Quine,. etc. in an analytic philosophy seminar this semester. Most of their most famous papers seem to me to have very serious problems. The same is true of influential books. Kant's Groundwork has serious problems. A Theory of Justice does too. Take any influential philosophy work you like, chances are it does too!

Consequently, I've tried to change my approach as a journal reviewer. I used to use something like the standard that many reviewers seem to use: namely, "Does this paper have serious problems? If so, reject!". Instead, more recently, I've tried to instead center the following two related questions in my mind:

1. Is this paper worth reading in the pages of the journal, whatever I think its problems may be?
2. Are whatever problems the paper has ones better addressed by follow-up articles rather than rejection or revise-and resubmit?

I have to confess that I've found this way of framing my task as a reviewer pretty transformative! Although I still reject papers and recommend revise-and-resubmits (with minor or major revisions), my anecdotal sense is that the above approach has made me more sympathetic to papers, leading me to be more inclined to recommend revise-or-resubmit or acceptance rather than rejection even if I think a paper has real problems.

The problem I face, though, is this...I now often feel a heck of a lot less sure of what editorial recommendation to make. I find it particularly difficult given that some journals explicitly contain instructions to reviewers to use "especially high" standards given the journal's submission numbers. Sometimes, my honest to goodness thoughts are something like this, "Here's my report, editors. Let me explain why I'm totally unsure what to recommend. Given that it's your journal, why don't you decide given the substance of my report and the other reports you receive?" However, although this actually used to be how referee reports appeared to work (earlier in my career, most of the reports I received didn't make a clear recommendation!), these days journals generally expect some kind of clear recommendation. Indeed, some of their websites make you do so (selecting either 'accept', 'reject', 'revise-and-resubmit', and so on). Consequently, in these kinds of cases, I'm no longer very certain what in the world I should be doing. To put the quandary a bit more precisely, I find myself facing the following:

1. Journals requesting the use of 'especially high standards'
2. Journals requiring clear recommendations in their online platforms.
3. My evolving intellectual conscience/self-conception as reviewer suggesting to me I should be less inclined to recommend rejecting interesting work with substantial problems. 
4. (1) and (3) together leading me not to have any clear idea how to satisfy the request in (2).

Anyone have any thoughts or insights on what a reviewer in my situation should do? 

Comments

[Trevor Hedberg]

I don't have a solution to this quandary, but I do want to clarify one thing about my prior post: I don't think any of the reasons for rejection that I mentioned are incompatible with adopting the "Does this add to the discussion?" standard. I think that all of the concerns I raise point toward ways in which a paper would fail to meaningfully add to the discussion around the issue. Most of my concerns involve presenting one's argument clearly, taking proper account of other people's research on the subject, and not misrepresenting your own conclusions or the views of the authors you're engaging with. Those seem like prerequisites to making a worthwhile contribution to the ongoing discussion (although I can imagine that not everyone will agree with my assessment on that point).

It's also possible for a paper to avoid all those problems and still leave me quite unconvinced that its conclusion is true. But in that case, the judgment should be "This is a good paper that I disagree with" -- not "I disagree with this paper so I'm going to reject it." I wouldn't advise rejecting a paper merely for the reason that you disagree with the paper's central conclusions.

[Marcus Arvan]

Hey Trevor: Thanks for chiming in. I didn't mean to suggest that your approach to reviewing isn't consistent with the "Does this add to the discussion?" standard. Rather, I meant to draw attention to the relative differences you and I seem to have in arriving at editorial recommendations.

You wrote in your post that you, "thought it would be very difficult to differentiate between papers that warranted R&R verdicts and those that should just be rejected. I was wrong...The reason I was wrong is that most papers I review feature glaring problems..."

I used to have no problem making editorial recommendations...when the standard I used was, "Does this paper have serious problems? If so, reject!" The problem is that in my case, the alternative standard ("Is the the paper good enough add to the conversation?") seems far, far more nebulous.

My point was that I am finding it increasingly difficult to differentiate between recommending rejection, R&R, or acceptance precisely because, when I adopt the latter approach, it's just not clear to me how serious enough a problem with a paper has to be for the paper to properly fall into one category or the other. In my own case, at least, the "Does this add to the discussion?" has made me *much* more tolerant than I previously was, and far less sure of what recommendation to make in any given case. That's what I'm struggling with.

[The other Marcus]

I think the "is this worth discussing" standard is right on, and to deal with the concern about journal standards, I think you can use something like Trevor's criteria:
1. For a paper to be worth discussing in a top journal, the argument must be very clear, have considered a greater deal of the relevant literature, etc.
2. For lower journals, not being as widely involved with the literature, or having (parts of) arguments that are not extremely clear and precise may be fine.

Alternatively (or alongside) you may also track journal quality by the "how worth discussing is this paper?" - i.e., does it have the potential to make a significant contribution in the relevant literature? Or does it focus on a smaller issue only interesting to a tiny sub-field? Both can be worth discussing, but only the former may be worth discussing in a top journal.

In terms of considering an R&R vs an acceptance (or minor revisions, etc.) you may think about the paper's consideration of objections. Obviously, for reasons initially suggested, we should not expect a paper to consider all possible objections - leave some of those to later papers. But, perhaps a paper meriting major revisions is one that has not considered a quite well-known or obvious objection.

I agree this is a hard thing to figure out, and I don't think (by and large) we can or should want an algorithmic process, but certainly having clearer criteria would help.

[Robert Zink]

Reviewers don't reject papers, they advise editors.

[Anon]

I particularly wonder about cases where one is asked to review an essay in which one's own work is criticized. I'm not sure how often this happens, but it seems to happen fairly frequently that, if Smith's theory of X is criticized in a paper, then Smith will be invited to review that paper. (At least it has happened to me that I have been asked to read papers where I am named and criticized, and other friends have confirmed having the same experience.)

In those cases, I really hope that reviewers don't hold to the standard of "is this convincing". In fact, I think that would be pretty bad professional practice. Right?

So, clearly in these cases, whether the paper is worth discussion is the better standard. Right?

[Assoc Prof]

I would imagine a kind of trade-off between (a) how much a paper is worth talking about and (b) how much a paper is well-researched and argumentatively refined.

If a paper is really, really worth talking about, then some argumentative omissions might be tolerable. Or if a paper is just moderately worth talking about, then we would want the paper to be very well-researched and refined in argumentation.

Since most journals are quite selective, I guess I usually look for a 9 or 10 out of 10 on both criteria to recommend acceptance. But if a paper is really extraordinary in one respect (an 11 out of 10, so to speak), then maybe a lower score in the other respect is OK.

[a philosopher]

"If a paper is really, really worth talking about, then some argumentative omissions might be tolerable."

What do you mean by "argumentative omissions"? As Marcus points out, *every* paper (even the best, most influential ones) contains conceptual confusions, unclear passages, and either invalid inferences or substantial undefended premises.

Take quite literally any paper you like in philosophy and stick half a dozen philosophers who work in that area in a room together with that paper for three hours. They *will* walk out having identified 2-3 major flaws and be at least slightly confused about what the author was even trying to say.

So it seems to me that "argumentative omissions" not only *might* be tolerable, but in fact *must* be tolerable, else nothing would ever get published.

Perhaps by "argumentative omission" you mean only *serious* issues, but then the issue is that two philosophers rarely agree on how to weight the argumentative, conceptual, and philosophical problems in a paper. Many times I've had someone read a paper and say "oh, well, you don't really discuss objection X, but that's not a big deal", only for the next person to take my lack of discussion of X to erase all interest in the paper.

I know I'm to some extent jumping all over you (Assoc Prof) for what was probably an off-hand remark. It's just that I think it's *really* important to push back as hard as one can against this standard that we should recommend reject unless a paper "convinces" us, or recommend reject whenever there are "argumentative omissions". In practice these are not enforceable or inter-referee consistent standards, and they easily devolve into ad hoc justifications for rejecting papers we simply don't like. (Since every paper has "argumentative omissions", there's always one there for your subconscious to find if you don't like it and want to recommend rejection.)

[Assoc Prof]

Yes, fair point a philosopher. I guess by "argumentative omission," I meant an omission of the kind that would normally warrant rejection (however we want to fill that in).

My point was that, in the case of work that is very insightful, creative, and "worth talking about," it could make sense to slightly lower one's expectations for argumentative rigor and thoroughness.