A reviewerly quadrilemma

Suppose you're asked to review an article for a journal on theory X, and you're competent with and know the literature on X. Next, suppose you think most of the literature on X is predicated on a particularly problematic philosophical method (e.g. intuition mongering), and that the only justified methodology in the area supports ~X (the methodology you support is consistent with a variety of other theories on the topic, but definitely not X). Suppose also that the methodology you find problematic (e.g. intuition mongering) has been criticized in the field (and pretty effectively refuted, in your own view), but many authors (including the author for the paper at hand) continue to use the method. Finally, suppose the paper you are asked to review defends a variant of X using the methodology you object to. What should you do?

Here are a few possible alternatives:

1. Decline to review the paper, on the basis that you cannot review it in an unbiased manner.
2. Review the paper, setting aside as far as you can your objections to the methodology the article presupposes.
3. Review the paper, making your objections to the methodology known to the author and editor, but otherwise set aside your objections to the methodology in making a recommendation.
4. Review the paper, taking the author to task for the problematic methodology they co-opt from the literature.

Which is the best option?

I'll lay my cards on the table. I'm sort of expecting commenters to say that (4) is the worst option--but why? Mightn't the failure of reviewers to contest problematic methodologies be one primary reason why problematic methodologies come to be so common in certain areas (i.e. reviewer after reviewer who likes the methodology accepts it, but those who object either decline to review or else review but don't impose their methodological preferences on others)? 

Next to (4), (3) seems to me the most honest approach--making one's objections known. But what is the point in simply making one's objections to the methodology known if author after author (including the one in question) continues using the method?

(2) seems to me the "most unbiased" option--yet why should one have to "bracket off" one's considered views about appropriate philosophical methodology (views that, again, others have argued for in the literature, as well)?

This leaves (1): simply declining the invitation. But that seems pretty weak to me too--deciding to simply leave papers with problematic methodologies to those who (potentially) buy into the method when, in your view, they really shouldn't (and, in your view, the continued dissemination of/support for the methodology is philosophically regrettable).

What to do? I don't but, but the above are my thoughts. What are yours? 

Comments

[anonymous]

I think you are missing:

5.) Review the paper, ask the author to address, (i) criticisms of the methodology extant in the literature, and (ii) the consonance of the author's own position, given that methodology.

I don't see why those aren't two perfectly reasonable considerations for reviewers to raise. It will be up to the editor of the journal, of course, whether the issues raised merit rejecting the paper, inviting it to be revised in the light of the criticisms, etc. I take it that the job of the reviewer is simply to make the author and editor aware of such potential problems and I don't see any problem with doing so in this case.

[Marcus Arvan]

Hi anonymous: Thanks for your comment! That's sort of what I was going for with (3), but the problem is, I'm assuming there's no way the author could adequately address the methodological issue in the paper (as the issue is at least an entire paper in its own right--indeed, an issue that many other papers have belabored in great detail). At best, the author could only address the issue in a cursory/dismissive way, which is precisely what (I'm supposing) one is tired of as a reviewer (e.g. one is a bit fed up with the literature citing the concern only to go on continuing to use the methodology).

[Derek Bowman]

What is the purpose - or what are the purposes - of peer review in philosophy?

Without at least a sketch of an answer to that question, I don't see how we can begin to answer the one that you've laid out.

If it's to help filter the content of journals to promote the advancement of knowledge, then surely publishing articles that rely on unreliable and discredited methodologies would be counter to that goal. Of course other philosophers - presumably including the one whose article you're reviewing - may well disagree about whether the methodology in question is unreliable and discredited. But you're being asked to review the article based on your expertise in the relevant field, so why shouldn't you use your judgment?

That seems to tell strongly in favor of option 4, though it might allow for something like option 1 or 3 in cases where you recognize that your assessment of the methodology is not shared by others you recognize as fellow experts, or by the editorial board of the journal.

Option 2 only makes sense if you assume that the purpose of peer review is something else; perhaps measuring some kind of procedural competence (or even excellence) using established philosophical techniques, irrespective of whether the methods used are likely to advance knowledge in the relevant areas.

This might be the appropriate standard if you thought either (i) there is no reliable knowledge to be had about the subject matter and/or (ii) the purpose of peer review in philosophy is more sociological than epistemic. For example, one social function of peer review is to determine employment and promotion within the field.

[Richard Yetter Chappell]

Hi Marcus, does it make a difference that the aspect of the paper you disagree with is it's *methodology*?

A generalized principle (4*) strikes me as pretty clearly wrong, if used to "take an author to task" for participating in a tradition (Kantianism, say) that the reviewer takes to be hopeless, and demonstrably so.

It's just not the place of reviewers to be making those kinds of judgments. Reviewers should judge whether a paper is a good exemplar of its kind, i.e. given the kind of project it is engaged in (which I think includes the kind of methodology used). It is utterly inappropriate for reviewers [who are not themselves here subject to the checks and balances of meta-peer review!] to (ab)use their powers to punish authors for pursuing projects of a type (incl. a methodological type) that they personally happen to disagree with.

No real good is achieved by this, even supposing that your goal of "purifying the literature" (so to speak) were a good one. If it's a good paper for what it is, then it will be accepted at a subsequent journal. All you're doing, by rejecting them now for your idiosyncratic reasons, is screwing the author around in the meantime.

In short: if you have objections to a whole style of philosophy, make them known in your own work (which may include papers, blog posts, etc.). Try to convince others with the force of your arguments; but use no other force than that. The peer review process is not the place for such blanket rejections. Articles for review should be assessed in their own terms, and not rejected just because of the type of philosophy that they engage in.

[Marcus Arvan]

Hi Richard: Thanks for your comment! I hope I didn't give the impression I actually favored (4), or would go with it. I share your concerns, and am more split between (1)-(3).

Still, I think (4) raises some interesting questions. Presumably, at some point it was appropriate, say, in physics for reviewers to stop taking seriously papers on the luminiferous aether; for academic psychologists to reject papers on Freudianism based on mere speculation; etc. Obviously, these are different cases than philosophy--as they are cases where basically everyone in the field came to recognize a particular methodology/theory as illegitimate (and philosophy, in general, is not quite like this). But still, I think it's an interesting question.

We are allowed as reviewers to "judge the arguments on their merits", right (that's supposed to be our job, no)? But, if we think an entire form of argument is terrible--and we think it has been shown terrible in the literature--why exactly shouldn't we be able to judge its merit in that respect? You suggest no real good would come of it, and that one should simply continue to argue for one's favored approach in one's own work. But suppose, to follow up on a previous issue (research and citation practices), a large body of literature hardly ever engages with the critique in question (this isn't the case on the example of intuition-mongering, but I can think of cases where this seems to me to have occurred). In that case, some good might come of rejections based on reviewers taking a certain methodology to task. Such rejections--if they started to be more common thanks to more reviewers rejecting the methodology--might lead to the members of the relevant literature to finally address the critique(s) in question, rather than continuing to ignore them. Which would be a good thing, no?

[Anonymous]

I think the answer to this depends on the answer to a couple more questions. Does the paper do anything substantially new or novel with this methodology? If so is the problem it addresses one that someone who does not subscribe to X, would recognize as a real problem? If the answer to those questions is "Yes," then I would be inclined to note my objections to the method and then put them aside as in 3. But if the paper makes a technical point that's only really important if you already buy this method in the first place, then I'd go for 4 and feel no guilt about doing so.
I'll give you an example of what I have in mind. If say I was given a paper that somehow used Frankfurt examples to establish a new and radical point about distributive justice, then I'd probably opt for 3 in reviewing it even though I think that Frankfurt examples are blatantly question begging. On the other hand, if it simply addressed an objection to Frankfurt examples (or worse an objection to an objection to an objection) then I'd opt for 4 and feel fine about doing it.

[Brad]

Marcus,
I think you should be discussing this issue with the editor of the journal who invited you to review the paper. The editor, author, and journal are entitled to confidentiality while you are handling their manuscript.
Most of your readers would be distressed to see you discussing THEIR own manuscript, which they sent to a journal, on this blog.

[Marcus Arvan]

Brad: I appreciate the concern, but the topic of this post is hypothetical, the questions it raises are general, and it does not discuss any paper/journal in any identifiable way.

[Joe]

I'm with Marcus. I am actually very confused about why anything other than (4) is the correct option, given the way the question is posed. How often are we forced to revise and re-revise our papers because we haven't cited and dealt with some relevant literature? All the time. Why does it matter that the relevant literature concerns our methodology, and not the move from P3* to C2?

I would also suggest that Richard Chappell is using language that makes this proposal seem less reasonable than it is. Traditions aren't methodologies: there is the discussion/evaluation of Kant, and then there is the method of deploying transcendental argument. If the method is flawed, it will infect anyone's transcendental argument, no matter what the topic. That's a huge problem, and since that method has been subject to many criticisms in the literature, at least a paragraph on those criticisms seems perfectly warranted (very odd for Chappell to call this "idiosyncratic", since that is by definition untrue of this situation).

He also assumes that (4) involves rejecting the paper... but clearly, a good reviewer can recognize that the paper would be outstanding were its methodology any good, and ask for a revision that defends the method. I don't see how this is any different from the usual gymnastic contortions we require of published authors.