Rethinking journal reviewing standards -- a rough first pass

As a follow-up to my previous post, in which I questioned whether dominant philosophical standards are exclusionary, in this post I will present a tentative, very rough first pass proposal for how journal refereeing norms might be revised so as to promote inclusivity, where this understood in terms of both of the goals Elisa mentioned in her post (including underrepresented people and underrepresented ideas). Again, just to be clear, I offer up the proposal only as a very rough first pass. My main hope is not that the proposal is adequate, but that it will generate further discussion of how we might improve things.

I want to begin by reflecting on how the review process typically works with philosophy journals. Almost all of the journals I have reviewed for give few, if any, guidelines for how referees are supposed to proceed. Basically, the paper is given to the referee, and the referee is to write up a review justifying a recommendaton (viz. accept, reject, R&R, etc.). Then, in light of referee comments, the editor makes an editorial decision.

The most striking thing about this practice is the sheer amount of latitude it gives reviewers (and editors) to judge papers by reference to their own preferences/biases. In my experience, referee comments tend to read as though the referee was primarily asking themselves something like the following questions:

1. Do I find this paper's argument persuasive?
2. Will the dominant readership of this journal find its argument persuasive?
3. Do the author's premises (A) cohere with "commonsense", as well as with (B) premises commonly appealed to in the literature in question?

These are natural enough questions to ask. The first thing I do when reading a paper is ask myself whether I think its argument is any good. But I think this is a problematic way for journal referees to approach their task, and for the reasons I outlined in my previous post. If most referees come from the same background (i.e. not only racial, gender, etc., but also philosophical background, viz. prioritizing "rigor" above all else), the practice of referees asking themselves what they think of the paper threatens to have unintended consequences: namely, most referees with end up rejecting things because the paper they are reviewing contradicts the "received wisdom" that they have internalized as philosophers. In other words, if a vast majority of reviewers come from one background X, and they tend to ask themselves, "Do I find this paper persuasive?", the end result will tend to be: rejecting papers that conflict with their background -- which I have suggested excludes alternative ideas and perspectives. And now notice how innocent it all is! Individual reviewers may not regard themselves as philosophically biased, and yet, if reviewers are given complete latitude to judge "for themselves" whether a paper is any good, the expected end result can be the systematic exclusion of underrepresented perspectives!

I think that this has happened in professional philosophy. Professional philosophers prize rigor, but unfortunately, they have also (as I argued in my previous post) defined rigor relative to the "commonsense intuitions" of an insular few. So, when someone with different intuitions -- or wanting to defend a different (e.g. feminist, racial, non-Western) conceptual framework -- the average reviewer at a mainstream journal thinks to themselves, "This doesn't sit with my intuitions or the intuitions of people who read this journal. Reject." I think this is bad. I think it leads to overly conservative, exclusionary philosophy. I almost don't want to say it, but I have a hard time reading some "top" journals these days. Some of them seem so narrowly focused, and certain debates so far down a deeply mistaken garden-path, that I find the work completely boring. There, I said it. :) This isn't to say that I think all work in analytic philosophy is boring, or even that most of it is. All I am saying is that too much of it is -- and does anyone really disagree with this? I hear it said all the time!

How, then, can philosophy become more inclusive and less boring? To get started, I think it is worth noting that a few journals present reviewers with a very different task than the traditional one I laid out above (viz. "Reviewer, decide for yourself, according to your own unguided judgment, whether this paper is any good!"). What is this alternative? Well, some journals (I can't think of which, but I've reviewed at least a paper or two for one!) instruct reviewers to rate the article under review along several dimensions. Very roughly -- and yes, I am paraphrasing very roughly off the top of my head, some journals give reviewers a form that reads something like this:

Rate the following paper on a scale from 1-10 (1 being "poor", 10 being "excellent"):

1. How clear is the paper's argument?
2. How rigorously is the paper's argument defended?
3. How innovative is the paper's argument?
4. Does the paper deal adequately with potential objections?
5. Etc.

I think this is good because it prompts the reviewer to rate the paper in question on matters that the reviewer might not have thought of or even care about but which would enable a good editor to prioritize things that the reviewer doesn't. Here are some additional questions that might be included so as to enable an editor to select articles that challenge conventional ways of thinking:

• How far does the paper challenge conventional thinking on the issue examined?
• Can this paper be expected to appeal primarily to a crowd sharing certain intuitions, but not appeal to people not sharing those intuitions?
• Can this paper's premises, even if they do not seem "plausible" to you, be reasonably expected to seem plausible to someone from a different background or vantage-point (e.g. feminists, non-Western philosophers, members of other underrepresented backgrounds)?
• Will this paper be remembered in five years as making a major contribution to the debate, or does it merely make a "small point"?

I don't know quite how these questions might be formulated -- and yes, I do realize that my attempt to formulate these has been pretty ham-handed -- but still, it seems to me a promising way to focus reviewer attention and judgments on criteria other than the conventional, conservative, potentially exclusionary ones they might otherwise be drawn to (i.e. their "own intuitions" and the "commonsense" of those similar to themselves).

If such questions/prompts were formulated well and then given differential weights by clever editors (for instance, since I think really innovative stuff is hard to pull off rigorously -- see e.g. Kant's Groundwork -- as an editor I might come up with a formula or something to discount rigorous work on "small problems" and prefer somewhat less rigorous work with particularly innovative insights), then, I think, we might have a mechanism for encouraging reviewers, and editors, to publish more inclusively (i.e. not excluding ideas/arguments simply because they do not appeal to a narrow in-crowd).

Anyway, although I'm not quite sure how to formulate such questions properly, that's the (very tentative) proposal. I'm curious to hear all of your thoughts!

Comments

[eyeyethink]

YES! I especially like your third and fourth bullet points. I'd suggest that you and I start a journal, if I thought anyone would take it seriously. (Two non-TT editors? WTF?)

I see the intent behind your second bullet, but isn't the answer always going to be 'yes'? Perhaps you mean to ask: "will the paper appeal only to those who accept mainstream intuitions?"

[Matt DeStefano]

Thanks for this post, Marcus, it's really interesting. I have some reservations about the bullet points. The third and fourth points seem to ask too much of any referee. When an editor queries for referees, I take it that they are asking for the referee's opinion about a piece. It would be a mistake, I think, to have those referees try to guess what philosophers with very different backgrounds or in very different areas might or might not find plausible about a given argument. Is it possible for a person to reliably predict what those with very different backgrounds will or will not find plausible? I'm skeptical.

If I were asked a question like this as a referee, I would be afraid that I would do more harm than good. Either by being too permissive and encouraging an editor to accept a bad paper ("I think this is completely implausible, but maybe some other philosophers may find it plausible"), or by thinking that those from different backgrounds would share my own notions of plausibility.

I don't know that I have a good alternative in mind (other than by pulling referees from very different backgrounds in the first place), but I thought I would share my concerns.

[Elisa Freschi]

Marcus, this is a very interesting post. Accordingly, the following ones are just suggestions:
As for me, when I review/ask someone to review an article, I try to check the "objective" soundness of the article first, i.e.,

—whether the overall structure of the article holds (no articles with no sections, or which repeat the same points again and again, or say that X without having shown why…)

—whether the author knows the literature on the topic s/he is dealing with. This is consistent with my general idea that one should read more (and also, I believe, with some posts of you, e.g.: http://philosopherscocoon.typepad.com/blog/2013/12/things-i-think-ive-learned-thing3-read-widely.html and, more relevantly: http://philosopherscocoon.typepad.com/blog/2014/02/on-citation-practices-in-philosophy.html). More in general, I mistrust authors who think they can come out with fantastic new theories and do not need to bother about reading anyone else's work.

I know, these are not completely "objective" criteria (as is evident from the fact that the lack of a quotation of X will be evalued very differently if you happen to be X!), but they still seem to me to be better than "Did you like the article?". Hopefully, they will force the reviewer to re-focus while reading the article.

This being said, I agree with most of your points.

[Moti Mizrahi]

Hi Marcus,

I like your proposal overall. But I have some doubts about the last bullet-point:

"Will this paper be remembered in five years as making a major contribution to the debate, or does it merely make a "small point"?"

First, as we discussed on the Cocoon before, it is difficult to distinguish between "small" and "big" points. For instance, one might think that Gettier’s point was "small" (after all, it’s just a counterexample against JTB), but this "small" point has had a great impact on epistemology. (Whether the impact was positive or negative is a different question.)

Second, what counts as a "major"—as opposed to a "minor" or "significant"—contribution to a debate?

Finally, I think that the "remembered in five years" condition sets a very high bar for publication. Many papers in philosophy "fall still-born from the press" (i.e., they’re not cited or even read), but that doesn’t necessarily mean that they are not worthy of publication. Maybe philosophers just don’t read widely enough. :)

[Marcus Arvan]

eyeyethink: I think we'd run a rad journal. :) But maybe the Journal of the American Philosophical Association got there first? (http://journals.cambridge.org/images/fileUpload/documents/J-APA_Editorial_Statement.pdf ) I guess we'll have to wait and see!

Matt: thanks for your comment. I understand the worry, but *I* don't think it asks too much. I read papers all the time where I think to myself, "I'm not sure if I buy these premises, but I expect there are people out there who *may* -- and the paper is interesting!" I think this kind of openmindedness is good, and I think too many reviewers (e.g. those who prioritize rigor above *all* else) seem to lack it. Moreover, I think one can recognize things that are likely to be attractive to people from other backgrounds and perspectives. When, for instance, I read Charles Mills -- who argues that traditional political philosophy, by all but ignoring race, ignores the reality that non-whites face (racial domination) -- I can recognize that although some traditionalists may not find the argument attractive, there may be a lot of people who will. And this isn't just an isolated case. Authors who come from different backgrounds and argue for different premises often make it *explicit* that they are trying to challenge the status quo from another vantage point. I don't think it takes a rocket scientist to pick up on it. What is *does* take is openmindedness to agree not to reject it simply because it "doesn't engage with the mainstream literature."

Look, to be sure, there is the possibility of a type 1 error (accepting a crappy paper). But, I say, current standards give rise to too many type-2 errors (rejecting *good* papers simply because the reviewer "finds it counterintuitive"). A good editorial process, I want to say -- just like a good judicial process -- has to strike the right balance between the two. And I think things right now have gone *way* too far in the conservative direction. Personally, I'd rather read a few more interesting-but-flawed papers than boring, unimaginative, but "rigorous" papers. Again, this may just be me, but others have complained about it too (and indeed, the new Journal of the American Philosophical Association recognizes it as a problem!).

Elisa: thanks for your comment. My reply is: absolutely! I was not suggesting that my bullet-pointed questions take the *place* of other questions (viz. "Is the paper clear?", "Is it coherent?", etc.). We have every reason to want those "objective" standards! My claim was merely that they should be supplemented with questions prompting reviewers to consider and judge the paper along other dimensions as well.

[Marcus Arvan]

Hi Moti: thanks for your comment. I take your point well. It's not obvious how to construe a "small" point vs. a "large" point. In one sense, Gettier's point was a small one, in another sense a large one.

I guess I would gloss the relevant distinction in terms of how much "conventional wisdom" the point overthrows. Yes, Gettier's paper was two pages long, but it essentially (in the eyes of many) refuted a conception of knowledge (knowledge=justified true belief) that had been taken more or less as given in Western philosophy ever since Plato. That's a pretty large point!

At any rate, that's how I think we should understand the distinction here. The more a paper buys into conventional wisdom, the "smaller" its point in the relevant sense. The more deeply it (successfully) challenges conventional wisdom, the "bigger" its point.

Finally, point taken on the "will this paper be remembered in 5 years" question. Perhaps it should be: "In your judgment, *should* this paper be remembered in 5 years?" That might deal with the problem of papers falling stillborn from the presses. Even if a paper *isn't* remembered in 5 years, it might be good for a reviewer to reflect on whether, in their judgment, it is the sort of thing that should be remembered.

[Giulia]

Thank you for the challenging post. I'm too young to be an experienced author or referee, but these points are definitely worth considering for the future. Not only editors (who can explicitly state such aims in the journal review practice rules) but also referees as individuals and the scientific community can and should think about the issues Marcus and and commentators raised.

I only have some problems with this guiding question:
Can this paper be expected to appeal primarily to a crowd sharing certain intuitions, but not appeal to people not sharing those intuitions?

Onestly, I don't think that being appealing in that sense is a crucial factor for the quality of a paper. I appreciate that we usually do evaluate a paper and its arguments also with respect to intuitions, but often a paper is strikingly good and philosophically interesting even if it is very counter-intuitive (whatever that may mean) and its thesis is prima facie paradoxical: many genuine contribution to philosophical debates were precisely like that, I dare say.
In point of fact, it's not even so important that the paper appeals to a minority sharing certain intuitions. While it's certainly desirable that the paper makes a novel point, and that a journal makes an effort to include non-mainstream approaches to a problem, the conformity to intuitions is not a reliable indicator of a paper's quality. A good philosopher should be able to defend her theory in a way that accounts for our intuitions (IF - a big if - there are such things), but it is not necessary to apparently agree with anyone's pre-theoretical intuitions, no matter if the intuitions at stake are commonsense, mainstream, or a minority's.

But I'm open to clarifications!

[Ruth Groff]

I love this post. I think that it is absolutely practical. I already review that way.

[John Jones]

As a post-graduate I gave a solution to the Analytic/Continental split, posing it as an organizational, and not a conceptual divide. This was radically different from the mainstream view, a view that was represented, in my case, by professor Christopher Norris. The professor did not discuss the ideas with me, deeming it sufficient to give my work a bare pass, even though the ideas were critically relevant to his own work. Naturally, I could say nothing for fear of compromising my academic ambitions. The rot starts well before the journal.