Should journals have clear reviewing standards?

I have received some amazing sets of comments from journal reviewers before. Some of them have been detailed and helpful, others detailed and critical. Whatever the reviewer thinks of my paper--whether they advocate acceptance, rejection, or whatever--as long as they provide a detailed report, I am satisfied. Indeed, I am even willing to take misreadings of my work with a grain of salt--as an indication, that is, that I could be clearer about something so that people don't misread what I've written. I have no trouble accepting criticism, providing the person giving it actually backs up their evaluation with philosophical substance. Yes, hearing that my argument is bad or unclear can be a little emotionally difficult, but, since we're doing philosophy, I appreciate the honesty. Indeed, I've felt--and expressed in published papers--a great deal of indebtedness to conscientious reviewers, as I know just how time-consuming it can be.

All that being said, like many other people (or so I've been told), I've also received my share of what I take to be totally irresponsible reviews, including a few recently that have really bugged me. I won't say anything about the reviews themselves above and beyond this: I don't think a one sentence review for a paper that has passed the desk-rejection phase is appropriate, nor do I think it is appropriate to advocate rejecting a paper on the basis of saying explicitly that you haven't bothered to read other work the paper is responding to or building upon. For the problem in both cases is the same: absence of philosophical substance. Advocating rejecting a paper on grounds of philosophical substance is one thing: it is fair. Advocating rejecting it without any substantive philosophical argument is another: it is unfair. For, unless the reviewer provides some philosophical analysis of why the paper the paper should be rejected, there are no grounds for thinking that the rejection tracks the merit of the paper at all, rather than the biases of the reader. 

Which brings me to a simple question: shouldn't journals have stated, reasonable standards for what counts as a legitimate review? Shouldn't they hold reviewers to some such standards, such as, "Reviews are expected to briefly summarize the paper's argument and provide some detailed philosophical rationale explaining how the paper's argument is successful or unsuccessful, not vague assertions such as 'this paper is not well-argued' or 'I haven't read the work this paper is referring to and doubt this journal's readers will have read it either'"? It sure seems to me that having such standards should be a standard editorial practice. An author who has spent months or years writing a paper and then waited weeks or months while their paper is under review--especially one that has passed the desk-rejection phase--deserves their paper to be subject to fair, responsible standards for review. So too, frankly, do those of us who, as reviewers, provide fair, detailed reviews, as it is hard to want to continue giving fair, 2-3 page reviews to other people's papers when your own papers are rejected on seemingly-arbitrary grounds, without any detailed rationale.

Or so say I. What say you? Should journals set and enforce clear standards for reviewers? If so, what should they be?

Comments

[Helen De Cruz]

AJP has a detailed set of instructions for referees http://aap.org.au/instructions
I like this bit at the end "It hardly needs saying that we would like referees to assess the paper, as far as possible, with regard to the quality of its argumentation, rather than in terms of the compatibility of its conclusions with their own positions and philosophical commitments. Writing a report is different from writing a reply."

[J]

Thanks for the interesting post! I agree with much of it: reviews should generally be philosophically substantive.

But I've myself recently become unsure of the truth of the following:

"Advocating rejecting a paper on grounds of philosophical substance is one thing: it is fair. Advocating rejecting it without any substantive philosophical argument is another: it is unfair."

First, are identifying substantial philosophical defects necessary for reasonably and fairly advocating rejecting a paper? I'm not so sure. Length, clarity, originality, and journal-fit could be sufficient grounds for advocating rejection. But these aren't necessarily philosophically substantive, are they?

Second, and more interestingly, I doubt that identifying substantial philosophical defects is sufficient for reasonably and fairly advocating rejection. Why? Because significantly flawed arguments might be publication-worthy. Most great philosophy articles, I would guess, have unsound (and profoundly mistaken) arguments at their core. Wouldn't you agree that there is plenty of room for reasonable disagreement about most important philosophical issues? If so, doesn't that show that it is unfair to advocate rejecting a paper merely because it is substantively philosophically flawed? (Maybe you and I are talking past each other here--am I misunderstanding what you mean by philosophical substance?)

When I review a paper I try not to regard my substantive disagreements with the author as reasons to advocate rejection (and I'd guess that you also try not to do this). That being said, I'm really not sure how to state my criteria for rejection...

[Clayton]

"Which brings me to a simple question: shouldn't journals have stated, reasonable standards for what counts as a legitimate review?"

Yes. This would make life easier for authors and referees. A simple rubric and some guidance about best practice from the APA would make life better for all of us by a significant margin.

[grad student]

I agree. And I think the lack of standards speaks against the profession. You don't need just competence to get a paper published, but you better be lucky too!

I recently sent a paper to a well-known journal in Britain.

The paper passed the first phase (Editor) but was rejected by the referee. I was shocked by the referee's comments though, since he/she was arguing against a view that I was not defending at all in my paper!

In an attempt to justify his/her argument, the referee even mentioned a very well-known philosopher as a reference. However, that philosopher never published on the topic of my paper! Which makes me think 1) the referee wasn't familiarized with the literature, and 2) he didn't read the paper.

[Marcus Arvan]

Hi J: Thanks for your comment! I didn't mean to suggest that one should reject a paper simply because one disagrees with it, or even just because one isn't convinced by the argument. As you note, the arguments many/most influential books and articles in the history of the discipline are typically messed up/unsound.

In fact, this is something I've talked about before. I tend to think philosophers in general, and especially reviewers, prize "rigor" over what are (in my view) much more important things, such as creativity and insight. (http://philosopherscocoon.typepad.com/blog/2012/05/has-contemporary-philosophy-over-fetishized-rigor.html ).

[Marcus Arvan]

Hi Helen: If only the first sentence of the AJP guide you quote were true (viz. "It hardly needs saying that..."). Apparently, it *does* need saying, as too many of us have repeatedly encountered the very behavior it says referees shouldn't engage in!

[Marcus Arvan]

Clayton: I agree. The APA should get on this. Journals shouldn't be able to use whatever editorial standards they wish, as they are critical gatekeepers for professional advancement. There should be a "best practices" statement on this by the APA, and journals should be held to it.

[Mr Journal]

Marcus,
The APA represents AMERICAN philosophers. Many journals are based in Europe; others are based in Australia; some elsewhere. The APA has no right to dictate anything to these journals; indeed, it is questionable whether they have a right to dictate anything to any but their own journal.

[Marcus Arvan]

Hi Mr. Journal: That's a fair point. But I'm not sure it follows from the fact that the APA only represents American philosophers that it has no right to use its considerable influence to affect what journals do. It is an association that *represents* American philisophers, including their interests in how journals are run, including journals in other places, in a manner broadly analogous to how states and nongovernmental organizations may influence foreign powers and organizations. Insofar as American philisophers are philosophers, it seems to me that we have every right to influence the discipline of philosophy broadly, not just in the U.S., through whatever representative bodies represent us, including the APA.

In any case, it is certainly within our rights as individuals and collectives of philosophers to exert pressure on journals to use responsible editorial practices, which was the main point of the post. I'll be curious to hear what others think about brining APA influence to bear on the issue.

[Former journal editor]

In fact there is an APA statement on best practices in journal publishing:

http://www.apaonline.org/?journalpractices

Perhaps it needs more publicity!

[Kenny]

Marcus, I think Mr. Journal was responding to your earlier statement, "Journals shouldn't be able to use whatever editorial standards they wish". I'm inclined to agree with him that the APA has no right to DICTATE editorial standards to journals in such a way that journals wouldn't BE ABLE TO use standards the APA doesn't like. But your comment in response seems perfectly correct: even if the APA can't/shouldn't dictate what journals (especially international journals) do, the APA should nevertheless use its considerable INFLUENCE to improve journal publishing, and should do this in a stronger way than it has to date. Founding JAPA and trying to use good editorial practices there is a good start on this. But there's a big difference between influencing and dictating.

[Richard Yetter Chappell]

I'm all for clear standards, especially ones emphasizing that a paper shouldn't be rejected just because the referee can think of a possible objection. (If the paper is subject to some very obvious, devastating, and unanswerable objection, then that's another matter. But very few objections are so serious as to suggest that the paper in question is not a valuable contribution to the literature. Rather, they merely suggest that further discussion -- say, in a response piece -- could be fruitful...)

I'm not sure about requiring summaries and "detailed philosophical rationales". There's an obvious trade-off here between providing value to those who submit to the journals, and being able to recruit sufficient referees. Given that journals in general have a much harder time attracting referees than they do attracting submissions, it would seem unwise for them to make the refereeing process any more burdensome than absolutely necessary. And, frankly, I think sometimes a very brief report in response to a clearly unsuitable paper is just fine.

I especially think it's fine to reject papers from top-5 journals on grounds of their being insufficiently ambitious / interesting. In such cases, I don't see what would be gained by requiring more than a couple of sentences explaining why this is so.

[Pierre]

Richard Yetter Chappell ---

It is conceivable that something would be gained, namely, that those whose papers are rejected will know precisely why their papers have been rejected. This is especially true, I believe, of young authors who do not yet have a publication record, and need to improve their writing skills on the basis of comments by their peers. If all they face is, as it were, a “black box” that issues a rejection, they will have fewer opportunities to improve the precise skills required to publish philosophical papers.

Two objections appear plausible: (a) Even young authors are assumed to be philosophically skilled; (b) Reading others’ papers should improve one’s own writing skills.

As regards (a), the basic assumption is that, when one grows into publication age, one has already written term papers, a master’s thesis, and so on. In short: she has already acquired the basic skills required to be a philosopher (or, if she has failed to do so, she should just abandon philosophy altogether). So why would that same person not be able to write a journal-worthy paper?

A possible reply is that the skills required to write a good term paper and those required to publish a good paper *tout court*, though not unrelated, are distinct. A term paper and a journal paper are not evaluated by the same standards, and one’s being able to satisfy “term-paper-standards” does not automatically entail that one is also able, *ipso facto*, to satisfy “journal-paper-standards.”

As regards (b), the basic assumption is that reading other people’s work is a key to understanding how this work makes its way to publication. This I take to be true, but not sufficient, in that one also needs to understand how a paper presents its problem, builds its argument, and so on.

Allow me a poor, but hopefully somewhat instructive, analogy. You may think of a house. Watching houses may help learn you what an appreciable house looks like, but in order to understand how it was built, you’d need to see the scaffoldings as well; and that’s precisely what you don’t get to see. Likewise, you don’t get to see the “scaffoldings” of a paper, which are, as it were, a matter of “backstage” involving editors and reviewers, but not you, the reader.

In that sense, there is something to be gained with more extensive reviews, at least for young authors: they could be quite helpful in improving those skills that depend crucially on the very context and practice of *publication*, and not only to the context of grading term papers.

[Marcus Arvan]

Hi Richard: My wife works in psychology, and has never once received either (A) a rejection without comments, or (B) a single review without detailed rationale. There are good reasons for this. The *point* of anonymized review is to obviate bias. Allowing reviewers to reject papers without detailed analysis invites bias, enabling them to provide editorial advice on the basis of nothing more than their "judgment"--judgment which may be biased in all kinds of arbitrary ways.