Do these things wreck anonymized review? If so, what should be done?

I was at a conference the other week, and had an interesting conversation with a really famous senior philosopher. I can't really remember how it came up, but we were talking about the sociology of the discipline, and he said something to the effect of how one's institution may affect one's publishing prospects. He specifically mentioned Rawls being at Harvard in this respect, but I can't quite recall exactly what he said. I think it was something to the effect of that Rawls' original 1957 paper "Justice as Fairness" probably wouldn't have been published in The Journal of Philosophy if Rawls hadn't been at Cornell and Harvard. I found this interesting at least in part because I've long been a bit puzzled by that very paper. In it Rawls, does not give much of an argument. In it, Rawls basically states his two principles of justice (pp. 653-5), why he thinks the original position is a good way to think about how they might be justified (pp. 655-6), claims (without any real argument) that it is "natural" to think that the parties to the original position would agree to his two principles (p. 656), admitting shortly thereafter that he hasn't offered any proof that the two principles really are the ones the parties would agree upon (p. 657). In the rest of the paper, Rawls talks about how justice as fairness embodies the notion of "fair play" (pp. 658-660) and how the theory is an alternative to utilitarianism (pp. 660-661).

Now, I love Rawls. I'm a Rawls scholar! But I have to admit I've always found the initial paper (and the longer version later published in Phil Review) puzzling, and precisely because Rawls seems to offer so little in the way of argument in it. Apparently, I'm not the only one puzzled. Rawls' paper was followed by a critical review, "Justice as Fairness: A Modernized Version of the Social Contract", whose author, Everett Hall, really takes Rawls to task for it! A few choice snippets:

I must be brief to the point of appearing harsh...So now we ask, "What is our author up to?"...He appears to be  giving us a little model of a society under narrowly specified conditions; if so, we can strip it of its appeal to the imagination and set it up as a small deductive system. Suppose, he says, such-and-such and such-and-such; then, logically, we would be justified in concluding a social agreement on certain principles of justice which he characterizes as principles of fairness in settling competing claims...

By so modernizing the [social contract] theory Mr. Rawls seems to have transformed one traditional objection into an anachronism. 'That objection was that, certainly for the most part, societies have not arisen from a social contract and hence that any justice applicable to them cannot be founded on it. But let us not be too hasty. In this crude form, as implying an actual gathering of the populace and an affixing of signatures, and so on, Mr. Rawls' social contract escapes censure, for clearly it has no such implication. But is it thereby freed from empirical assumptions which may as a fact be false and in any case have not been verified? 

That Mr. Rawls does not suppose he has given us a little social geometry is suggested by the language in which he draws his conclusion: "Given all these conditions as described, itwould be natural if the persons were to accept the principle of justice previously discussed." And reading on we find sueh expresions as "each might consider it reasonable, . . ," "these remarks are not . . . offered as a proof. . .-." Now the use of such phrases may indicate no more than that he considers his conclusion compatible with his premises (i.e., that supposing parties in a society were to get together under the conditions and motivations stated, there would be no contradiction involved were one to go on to suppose that they would accept Mr. Rawls' principles of justice). But this would constitute a weak authentication indeed... 

A time-worn objection to the social contract theory thus still appears relevant. Social institutions do not seem to have arisen in the way that theory states nor to review their practices in a fashion consonant with it.

Contemporary readers will recognize literally all of the criticisms Hall is raising: (1) that Rawls' model is question-begging (he basically helps himself to the idea that we should treat people fairly, and the original position is little more than a teasing out of that supposition), (2) Rawls is focusing on ideal theory ("a model society" satisfying very narrow conditions) to the complete exclusion of very important and relevant nonideal facts, (3) Rawls appeals a number of empirical assumptions without substantiation, (4) Rawls' argument for his two principles of justice seems flimsy, etc.

This is not to say that all of these criticisms are sound -- although it is worth noting that many people think he never really came to terms with any of them (criticisms that Hall raised to Rawls' very first article on justice!). But regardless, it is striking to me (and apparently to Hall, and to my fellow conversant the other week) that Rawls got away with all this. We are accustomed, of course, to the idea that "great philosophy" is really rigorous...but Rawls' initial paper seems to fall far short of any plausible standard of rigor. Rawls doesn't provide an argument so much as he just outlines a new way of thinking about justice. And, as interesting as that theory ultimately turned out to be, Hall raising a lot of obvious and serious problems for the nascent theory, and Rawls addresses none of them. And yet..."Justice as Fairnes" nevertheless landed in the Journal of Philosophy.

Anyway, why am I going on about this? Well, because, again, my famous senior conversant was suggesting -- not implausibly, in my view -- that the system of peer-review isn't exactly fair (pardon the pun!); that there are all kinds of sociological phenomena at play in the publishing game. Such as? What are some examples of phenomena that might affect the outcome of "anonymous peer-review"? Some of these have been noted before on this blog: so-called "Google reviewing" is one issue (referees looking up paper-titles on Google). I know that this has happened to me several times. I have analytic software on my homepage and academia.edu page, and I've gone months without anyone Googling a given paper title and then getting several google hits on the paper title a day or two after submitting that paper for review. This is just one issue. Here's another: people getting invited to give colloquia or presenting at multiple conferences. I know some people, for instance, who got really plum jobs right out of grad school. Before they had published anything, I saw that they were giving colloquia talks at top-25 R1's. Good for them, I thought! But wait, doesn't that compromise anonymous peer-review? Isn't it likely that if they send something to Phil Review, the paper we be sent to someone who already saw the paper given as a talk? And what about conferences? The adjunct who cannot afford to attend conferences is, it seemed to me, at an immense disadvantage. Or indeed, what about blogging? Those of us here at the Cocoon and other blogs have a pretty visible public footprint. Isn't it likely that journal reviewers will know what we are working on?

The ways in which peer-review can be (and probably are) compromised are legion. But, what can we do about it? Should editors be more invasive about making sure referees don't know who a paper's author is (viz. by asking the referee to disclose any prior familiarity with the paper)? I don't have any clearly good answers to these questions. I've suggested an alternative peer-review model before: roughly. the kind of public (rather than anonymized) review process used in physics. But, that proposal got very little traction. Anyway, I guess I wanted to write on this stuff because I thought it might be worth discussing. What do you all think?

Comments

[Daniel]

I don't think it's a full answer, but asking the referee to disclose any prior familiarity with a paper is pretty common now, and I think it's a good practice.

[Justin Coates]

Journal of Philosophy used to have a policy of publishing symposium papers from the Eastern APA. Those are the longer (~5000 words) papers that typically have one or two very substantive replies. So one shouldn't be surprised that under-developed ideas were sometimes published in JPhil (though, it's my understanding that JPhil eventually did away with this practice because too many of the papers were underdeveloped).

Now you might think that the APA wouldn't have accepted Rawls' paper if he hadn't been a hot shot from the Ivy League. I can't speak to that. For what it's worth, Mortimer Kadish and Wilfrid Sellars also had symposium papers published in that issue. I'm not sure where Kadish was at, but I think that Sellars was at Minnesota during the period in question. In any case, a quick look at the folks published in the most recent issues of JPhil and Phil Review reveals a lot of names that I don't recognize at all and a lot of institutional affiliations that don't appear at the top of the Leiter rankings. This warms my heart at least. No doubt, you're right to worry that some people Google to see who they're refereeing, but all in all, I think that things are so much different now than they were in the 50s that I'm not sure why Rawls' paper is supposed to be an issue. Its inclusion in an issue of JPhil strikes me as not having much to do with the real issues in fairness that exist today (some of which you rightly touch on). Unfortunately, like you, I'm not sure what might be the best solution to our current problems.

[Marcus Arvan]

Hi Daniel: I think that's a good convention, but in my experience as a reviewer there is very little policing of this. I cannot recall, for instance, ever having to read and (say) check off a *box* attesting to the fact that I had no familiarity with the paper (a box of the sort that authors must check off in manuscript-submission programs to attest that their work is not under review elsewhere). Making reviewers read and check off such a box might be one small way to better police the issue.

And, how about this as another? How about requiring authors to submit a list of *universities* they have presented their paper at? This might sound "crazy", but I don't think it is very onerous (who doesn't keep a list of their presentations on their CV), and it could give editors reasons not to send the paper to people at places the person presented their paper. This, I think, would be a very good change.

[Olle Blomberg]

Given that people now post drafts on PhilPapers, academia.edu and other web sites (which is surely a good thing, people should do this without worrying about compromising anonymous peer review), I suspect it is often virtually impossible to keep the author's identity hidden from the referees. And it seems to me that keeping the author's identity a secret is relatively minor factor in deciding who are best placed to review a paper (avoiding conflicts of interest is sure important though). A good policy would be to always request referees to state whether they know the author's identity (as well as state any potential conflict of interest) when they reply to a request to review. This can only help editors make informed decisions about whether to publish. Speaking of which, if it is important that reviewers don't know the identity of the author, isn't it just as important to keep the identity of the author hidden from the editor (who, after all, is the person making the decision about whether to publish or not).

Anonymous peer review is not just about keeping the author's identity hidden from the reviewers but also about keeping the reviewers of the reviewers hidden from the author. I think this is just as (if not more) important for avoiding the peer review process to become (even more?) unfair. If you write a review of a paper written by someone at a prestigous/powerful institution, and the author knows your identity, then there is potentially an incentive for you to write a more favourable review.

[Olle Blomberg]

Correction: First sentence of the second paragraph in my last comment was (of course) supposed to state that anonymous peer review is also about keeping the *identity* of the referees/reviewers hidden from the author.

[Mark van Roojen]

FWIW, I think that it *is* important to keep reviewers from knowing who wrote the papers they review if at all possible. More important than the other way around. In my experience, not knowing that someone whom I respect did not write the paper under review is important to incentivizing giving that paper a fair shot. So Googling authors is very bad and I'm frankly surprised that people can't keep themselves from doing it. I understand the curiosity, but if you don't have the self-control not to do that, how do you function when facing other temptations? I'm wondering what contributes to the seemingly widespread violation of this norm. It seems like it needs an explanation.

I think most journals at least ask for disclosure of such knowledge. I always let the editor know upon being asked to review, though I also generally let the editor decide whether I should continue as referee. I don't think it worth adding information about where one has given the paper to the mix of things an author must disclose. Asking people whether they know who wrote the paper is probably sufficient and much less work. I do think it would be better if journals tried a bit harder to find people who didn't know who wrote a paper. Interestingly, I've also found that I am often wrong when I think I know who wrote a paper.

[Olle Blomberg]

I've never been asked to disclose whether I know the author's identity when I've been requested to review a paper, although I have disclosed such knowledge when I've had it. (Admittedly, I can count the papers I've with my fingers, so this may not mean much.) That should be the norm I think. Also, perhaps journals should explicitly request reviewers to agree not to try to find out the author's identity when they agree to review? This might be sufficient to enable (some) reviewers to withstand the temptations of curiosity as well as clarify that it would be a norm-violation.

I suppose someone might try to find out the identity of the author for a reason beyond pure curiosity if they strongly suspect who the author is and want to disconfirm this suspicion in order to rule out a conflict of interest. (Of course, there are better ways of doing that, such as directly asking the person they suspect wrote the paper, or raising the issue with the editor.)

[Mark van Roojen]

Olle's Idea of asking people to agree not to contact seems like a good one to me. Many of us are better at keeping most promises than at complying with tacit norms unaided. Funny how little things can make a difference.

[Marcus Arvan]

Olle: I completely agree. Every journal editor should engage in the practice to enforce the norm.

Mark: I agree with your agreement! :)