Good but not good enough

Commenting on my Peer Disagreement & Peer Review post, Joe writes:

The external referee should judge whether the manuscript is worthy of publication (E.g., Is there an argument? Does the conclusion follow from its premises? Etc.)

Now, if manuscripts are deemed publishable or not based on the quality of their argument, then what could a referee mean by saying "This paper is good but not good enough to be published in Prestigious Journal"? (I've received this sort of comment from referees on several occasions.)

Presumably, if a paper is good, then it makes a good argument (i.e., the conclusion follows from the premises and the premises are true/probable/plausible). If that's the case, then on what grounds can such a paper be said to be "good but not good enough for Prestigious Journal"?

Comments

[Daniel]

This seems pretty straightforward to me. Even among papers with no obvious errors (e.g., as you put it "the conclusion follows fro the premises and the premises are true/probable/plausible"), there are lots of important distinctions to be made, especially as regards originality/ambition. Imagine a paper that convincingly establishes a conclusion, but where that conclusion either isn't very surprising, or is quite limited in scope. Such a paper might be rejected (with good reason) at the most selective journals, and accepted (again, with good reason) at less selective journals.

Granted, judgments about the ambition/interestingness/originality of a paper are more subjective than judgments as to whether the main argument of a paper is valid, or even than judgments as to whether the premises of the main argument of a paper are true/probable/plausible. But I don't see how one can maintain that referees should never make them, or never use them as grounds for decisions about whether to recommend accepting an article for publication. Take the best undergraduate paper you've ever graded. Maybe you gave it an A, or an A+. I doubt you'd recommend publishing it in the most selective journals int he profession (or maybe you have come across one such undergrad paper--in that case, pick another one), and I suspect your reasons would be the sort of reasons that I mention above.

[Marcus Arvan]

Moti: I agree with Dan here. I don't think it's good enough for an argument to be good. It's more important for an argument to be *interesting.* Don't believe me? How many good arguments are there in Kant's entire corpus? By my lights, few to none. Just about all of the arguments fail. But they show fantastically new, inventive, deep, and complex ways things of looking at things. By my lights, most of the greats are like this. I know from many of your posts that you (like many philosophers) put pride of place on good arguments. I, frankly, do not. I would rather have an interesting-but-*wrong* arguments eight days a week than a good but uninteresting argument.

[Joe]

I definitely agree with both Marcus (if I may) and Daniel (if I may). (Also, let me say here that I very much appreciate the discussion you and Marcus had in the comments on the original post.) There are lots of factors that contribute to the referee's judging a manuscript to be "publishable."

One factor not yet mentioned by Marcus or Daniel concerns the editor's interest in seeing more work published in one area of philosophy rather than another. The editor may prefer or find more interesting articles in one particular area. Authors have to understand that manuscripts sometimes just shouldn't be submitted to journal x because that journal doesn't publish manuscripts in that area of philosophy. An article, e.g., on modal realism ought not be sent to Aperion.

This discussion reminds me of a controversy had among Collingwood scholars (if memory serves me correctly) over whether some editors of Mind unjustifiably rejected articles he submitted to the journal. The claim was that Ryle, when he took over as editor of Mind after Moore, rejected any manuscript that supported idealism or other fringe views, i.e., Meinongianism. Because Collingwood was an idealist, his manuscripts were rejected when they were submitted to Mind. Some anecdotal evidence exists that Ryle said in conversation and in correspondence he would not seek publications of manuscripts in some areas of philosophy. But Warnock's article on Ryle's editorship appearing in the centenary issue of Mind (circa 1976) argues that Ryle did not discriminate against articles in particular areas of philosophy, though he might have been well within his power as editor-in-chief to do so. Warnock points out a few examples of articles published under Ryle's guidance that advocated idealism.

(Reading that centenary issue might be helpful to you.)

Also, I know that Marcus mentioned Thom Brooks in a comment on your original post. I highly recommend two of Thom's papers he has posted on SSRN. One is "The Academic Journal Editor - Secretes Revealed," and the other is "Guidelines on How to Referee." I believe you would find both of these articles helpful in navigating the terrain.

[Moti Mizrahi]

Thanks very much for the comments, everyone.

I hope you will bear with me as I raise additional questions.

Daniel,

It seems to me that the problem with judgments about ambition/interestingness/originality is that they are not only subjective but also relative. Take interestingness, for example. When a referee judges that a paper is not interesting, does s/he mean that *s/he* finds it uninteresting or that the journal’s *readership* will find it uninteresting? If the former, is the fact that one referee (or even two) finds a paper uninteresting a sufficient reason to reject that paper? If the latter, what are the referee’s grounds for such a judgment? The referee hasn’t surveyed the readership, I assume.

Marcus,

I am not a Kant scholar, so I will take your word for it. So let’s assume that Kant’s arguments are bad.

Now, suppose that Kant is our contemporary and that he is trying to publish his core ideas in the most prestigious journals. Do you think that he would be successful?

Joe,

Thanks very much for the references.

Brooks talks about the following standards of “publishability”: venue (e.g., book review, discussion note, article, etc.), fit (e.g., a paper on scientific realism is not a good fit for a journal like Ethics), and contribution (a specialized narrow topic as opposed to a more general topic).

Please correct me if I am wrong, but I thought that, once an editor gives a referee a manuscript to review, the editor has signaled to the referee that this is a paper s/he can see being published in his/her journal (otherwise, why waste the referee's time?). So the questions of venue, fit, and contribution are sort of settled already. Now the editor wants to know if the paper makes a good case for its conclusion, which is why s/he needs the services of a referee who is well-versed in the relevant literature. Do referees also make judgments about venue, fit, and contribution? If so, should they? Or should these judgments be made by editors?

[Marcus Arvan]

Moti: I'd like to respond to your comments to Daniel, given that they apply just as well to what I wrote.

You write, "It seems to me that the problem with judgments about ambition/interestingness/originality is that they are not only subjective but also relative."

I flatly deny this. I think Kant is objectively more interesting than Average Joe Philosopher. You will of course now ask what I mean by "objectively." We can debate that if you want, but for the time being I'll just say that I don't know anyone who thinks it is purely subjective or relative to consider Kant more interesting or original than most philosophers. Kant is more interesting and original than most other philosophers.

As to your question for me, would Kant have gotten published in Prestigious Journal if he were alive today? Hard to say. It's hard to recognize genius when it's staring you in the face, in large part because geniuses are so different than the rest of us. Anyway, who knows. I never claimed that journal reviewers or editors are excellent at recognizing genius. I only meant to suggest that judgments of originality play a very large role in editorial decisions. As a matter of fact, I think accurate judgments of originality are very difficult to make...but that does not make them subjective, any more than it is difficult to judge the correctness of a very difficult proof in mathematics.

[Marcus Arvan]

Moti (and others): sincerest apologies if the tone of my comments the past few days has lapsed into a sense of annoyance. I'll put a lid on it. Been upset for a few days for reasons not worth going into. Seems my upset is bleeding through here. Probably a good idea for me to stay away from commenting for a couple of days. Anyway, sorry...as head moderator of this blog I of all people should be concerned with keeping things supportive. See you all in a few days, after the emotions have (hopefully) passed.

[Moti Mizrahi]

My question about Kant came up as I was reading Brooks’ “The Academic Journal Editor—Secrets Revealed.” (Thanks to Joe for the reference.) Brooks writes:

“I paid careful attention to the difference in argumentative strategies adopted by successful and unsuccessful submissions. The most successful submissions were often the best in appearance, clearly structured, focussed on one or two central points, and the paper would be strictly limited to its topic. The less successful were often poorly presented where its central contribution was left unclear or unspecified and these papers might include many points, footnotes, and sometimes whole sections that were unnecessary or even off-topic.”

If Brooks is right about this, then it seems that a contemporary referee reading Kant might judge that Kant’s writing is “all over the place,” as opposed to being “focused on one or two central points” and “strictly limited to its topic.” Would you agree?

[Daniel]

Moti: I'm not sure whether I agree that interestingness is relative. If you're using a non-normatively characterized dispositional notion of interestingness (i.e., something is interesting just in case it interests some non-normatively characterized population), then sure. Though a normatively characterized one (i.e., something is interesting just in case it ought to interest some population, or just in case it interests a population of *good* judges), then establishing the relativity of interestingness is trickier. And I suspect people often have at least partially normative characterizations in mind when they talk about whether a paper/argument is interesting. I suspect that's what Marcus is up to above.

Maybe you're skeptical about the practical applicability of any normatively characterized notion of interestingness; maybe you think nothing is interesting in any normatively loaded sense, or that we ought to be skeptical about our ability to tell what is interesting in such senses. Even so, I don't see how you can avoid thinking that sometimes, appealing to some notion of interestingness (maybe a relative one) is necessary to decide why a paper shouldn't be accepted. My example about excellent undergrad papers wasn't a throwaway point, and I'm still curious to know what you think about it, if you think that these "mushier" criteria (interest/novelty/originality) should never be used in decisions about which papers to accept.

[Brad Cokelet]

Here is a variant on Daniel's point: experts can judge what does and does not substantively advance a debate or raise new issues that have not been discussed in the literature. I also think they are well-placed to judge which conclusions have potentially wide-ranging implications and which don't. I think a good referee will pass on these judgements to an editor and be candid about whether they are brute judgements or ones that can be backed up with explanations and examples.

[Moti Mizrahi]

Daniel: Your remarks about a normative characterization of interestingness seem very sensible to me. As you rightly pointed out, however, I have some doubts about our ability to tell what is interesting in this normative sense. Of course, I am not denying that we make this sort of judgments. But the fact that we use these “mushy” criteria to judge what is interesting and what is not doesn’t mean that we should. So I think that we need be able to say exactly what we mean by “interesting” in this normative sense, before we can say whether or not these “mushy” criteria should be used to make decisions about manuscripts.

[Joe]

Hello Moti: (I've been traveling again, so I'm catching up with the comments, sloooowly) Let me respond to your concerns of 7/26 10:18am.

First, is passing on a manuscript to an external referee a signal that the editor (and, perhaps, the editorial board) can see the ms being published in the journal? Passing on a ms to external referees should not be interpreted as an endorsement, in any way, of a ms. Let's face it: editors are not omniscient. If it signals anything, it signals that the editors seek expert advice on an area with which they are unfamiliar. I take it that the purpose of the external referee is to give the editorial board some professional, expert guidance for what should be published or what should not be published in an issue of their journal.

Second, should referees be making judgments about venue, fit, and contribution? Sure, presuming that the external referees have an expertise in a sub-field consistent with the thesis of the ms and knows what other journals are out there. The referee will know what journal the ms might fit. So, if the referee doesn't believe that the ms belongs at x journal but might fit better at y journal, then that might be something the referee should at least hint at in the referee report. In fact, many do. When I review a referee report, I look for technical references. Whenever a referee inserts a technical reference or says things that might show the ms wasn't built for a general audience, they're signaling to the journal editor that the ms might fit better at a technical journal (Think here of something like the difference between Jrnl of His Phil vs. Ancient Philosophy).

Finally, should these sorts of judgments be made by editors? Editors might not have the ability to make these kinds of judgments because they're unfamiliar with that particular sub-discipline. So, in some ways, they depend on the referees to make judgments about venue, fit, and contribution. Ultimately, it is the decision of the editors to publish a ms, but having reliable, competent external referees is a huge asset for any journal editor.

I hope this has been clear and at least somewhat helpful.

[Moti Mizrahi]

Your comments are very helpful, Joe. Thanks!

I thought (wrongly, I guess) that the editor would be in a better position to make judgments about fit, since the editor is more familiar with the journal, its mission, its readership, etc., than any referee.

[Joe]

Oh, no, I don't believe you were wrong to think that… Let me clarify what I said in my coment of 7/28 (203pm).

The editor of x journal knows what fits best in x journal. But external referees, more familiar with the manuscript's sub-discipline, might recommend publication of some manuscript in y journal because its thesis fits y journal better than x journal. The editor of x journal doesn't know y journal or whether the submitted manuscript fits y journal. So, in this way, the editor depends on the external referee's judgment of fit for another journal.

[Moti Mizrahi]

Thanks very much for the clarification, Joe.