The Veil of Anonymity

There has been some discussion recently, both on the Cocoon (also here) and elsewhere, about how some referees use Google to discover the identity of authors whose papers they are reviewing.

As Joshua A. Miller nicely puts it, some think that more blindness will help address this problem (e.g., triple-blind review), whereas others think that less blindness is better. But isn’t it blindness that is at the root of the problem to begin with? That is, the veil of anonymity gives reviewers license to google titles and excerpts from papers. After all, who’s gonna know?

In that case, what if we remove the veil of anonymity completely and hold reviewers accountable? If the identity of referees is known to both editors and authors, would referees do a better job at reviewing papers? What do you think?

Comments

[Anonomish]

The system is flawed. In some areas of philosophy, everyone knows almost everyone, such that blind refereeing is only possible if the paper is by an outsider. In the areas where this happens (and where I know about it), I think it harms the areas as a whole. Discussion is driven in the ways it is, for the wrong reasons.

That said, I think biases in favor of famous folks, and against non-famous folks, are too strong. So I'd cast a vote in favor of triple-blind, as opposed to non-anonymity.

[Clayton]

Speaking just for myself, I would referee less (and would have refereed much less) if I couldn't review things anonymously because of the protection it affords me.

[Tuomas]

It's common practice in many sciences that refereeing is not blinded at all. There's an obvious risk here: the work of famous people might be accepted without any regard to quality, whereas there might be a bias against those from less distinguished institutions etc. Since these seem to be problems regardless of anonymous refereeing, it might not make that much difference in the end. I'm rather ambivalent about the whole issue. What I think is a much bigger problem is that refereeing is so slow -- why do referees sometimes sit on papers for six months or more?

[Andrew]

I'm with Clayton on this one. Lots of referee work is done by younger folks--especially by those of us who are professionally active and who answer emails in a tmely fashion. But such younger folks often have more to lose from, say retaliation from irate authors. I referee a lot these days--at least a paper or two a month. But I doubt I'd referee at all if I weren't protected from such retaliation by anonymity.

[Moti Mizrahi]

Thanks for the comments, everyone.

I can understand the worry about retaliation. As Tuomas pointed out, however, many scientific disciplines do not practice blind refereeing at all (e.g., physics). But I suppose that in those disciplines, too, juniors get to review the work of seniors. So I wonder how they deal with the threat of retaliation. Does anyone know?

[Marcus Arvan]

Moti: My (admittedly anecdotal) impression is that there isn't a serious problem with retaliation in physics -- the reason being that online debate about new papers on discussion-boards is carried on my so many different people, both by bigwigs and small-frys, that unless someone really sticks their neck out and acts like a real jerk, it is hard to single out any particular person for retaliation. My understanding is that an online consensus about the quality/importance of a paper is usually arrived at pretty quickly, given that when a paper has serious problems, many different people simultaneously point them out. Finally, although the system of course lets in bias (in favor of "famous" people), my impression is that it's a less biased system overall than ours, since in our case corrupt reviewers operate in secrecy, while in the physics case unfair online reviews -- including those that are too easy on "famous" people -- are often (I'd say, typically) called out by others and *shown* to be unfair, thus counteracting the bias better than in our system of "blind" review.

[Anon87]

I think we are going about it wrong. Why is it that we are trying to go out of our ways to add more (or less) security for such procedures, instead of attacking the issue of referees not living up to a basic standard of professional conduct. at the end of the day it is the people who ignore this blind-reviewing procedure, so they can peak behind the veil, who are acting in an unprofessional way. i think it's a culture of arrogance and misplaced pride that needs to be weeded out.

[Marcus Arvan]

Anon87: Well, one obvious problem it that, in our present system, it's neigh impossible to enforce the standards. It's one thing to say something is wrong; it's another thing entirely to actually do the right thing when they think they can get *away* with doing what's wrong.

This, in a nutshell, is why I prefer the approach to peer review in the hard sciences (especially, physics). Online public discussion amongst one's peers is by no means a bias-free system. That being said, at least in a public forum biases can be challenged and brought to light -- and, I would say, typically are. In physics, if someone thinks a paper by a "small fry" from a no-name school is being reviewed unfairly, one can make a *case* for it in public. Similarly, if people think a "bigwig" is getting off too easy, there again people can make a public case for it. This process isn't foolproof by any means, but I think there's a case to be made for it minimizing bias far better than our present process: where individual reviewers get to evaluate papers more or less however they like, "google reviewing" or not even *reading* the darn paper carefully or charitable, without much accountability. At least in a public forum, a paper's supporters and detractors can both call each other out for (A) bias, (b) uncharitable and careless reading, etc., arriving at a more genuine *peer* evaluation (by many peers, not just one to three random reviewers) of the paper's merits.

[Tuomas]

I think that Marcus is on to something here -- and I now see that removing blind review might even help with the appalling review times, since no one would want to get a reputation of sitting on papers for months and months. Well, at least it might help if it would still be possible to find qualified reviewers to do the job. But it does seem true that the veil of anonymity enables misconduct and it's the type of misconduct that's generally harmful for the early career folks in particular.

The one important difference between philosophy and, say, physics, is that there are so many fewer people involved in the first place. In philosophy you very often know the people whose papers you might be reviewing personally (which obviously complicates matters), whereas in physics that's bound to be less common, even if not entirely atypical.

[Helen De Cruz]

As I mentioned over at the NewApps post on this (in the comments) another crucial difference between philosophy and physics (or most other disciplines) is that philosophy journals have very low rates of acceptance. A top philosophy journal has a typical acceptance rate of < 5%. To compare, Science and Nature have acceptance rates in the ballpark of 6-8%. Baseline probabilities: it's comparably easier to get into Science than into Phil Review or even some good specialist philosophy journals (I seem to remember JHP having a less than 5% overall acceptance rate).
In physics, a very respectable journal like Physical Review Letters has a whopping acceptance rate of 35% (according to their website). This journal is usually listed in the top 10 of physics journals. Imagine a journal of the calibre of Phil Studies or Analysis with that kind of acceptance rate.
So, given the very limited space we have in philosophy journals, and given prestige bias, we should strive for more anonymity, not less. Triple blind refereeing is the way to go. Even if that isn't feasible or the norm in other disciplines, that does not mean philosophy should abandon the quest for more anonymity, and hence our best shot at fairness, in refereeing.

[David Morrow]

From what Marcus is saying, I think it's hard to draw any lessons about piecemeal reform from the physics case. The public online vetting process transforms the formal review process. It might account, for instance, for the high acceptance rates that Helen mentions: Physicists probably get clear signals in advance that their work won't make it into Physical Review Letters before they decide whether to submit. And reviewers' job, presumably, is simply to summarize the public consensus.

So when it comes to philosophy, it seems to me that unless we totally switched to something like the physicists' system, making the system less blind would make things worse -- especially because of the strategic effects that Clayton explained.

[Lewis Powell]

I think people are far too skeptical about the possibility of a culture shift in attitudes towards blind reviewing and refereeing making a difference.

We haven't even *attempted* any sort of concerted effort towards discouraging referees from googling the authors of papers they review (say, by stressing the importance of blind review when securing referees, and having them separately affirm their blindness when they submit their reports).

It is not always possible to design systems that cannot be "gamed" by participants who aim to undermine the system. That does not mean we should abandon all such systems, because it is possible that they could be used wrong. It means you attempt to use mechanisms outside the formal structure of that system to encourage people to follow the spirit of the system.

The simple fact is that there is a lot of evidence that people in general are very bad at controlling for the sorts of biases that enter the picture when they know the identity of the person whose work they are reviewing. It seems to me that philosophers are far too confident that they can remain impartial, given the empirical evidence about implicit bias. Moving away from a blind review system is pretty clearly going to disadvantage people whose names are coded in particular ways with respect to gender and ethnicity, not to mention the advantage it gives to people who are already well known in the field.

We're talking about studies where identical resumes are sent in, differing only in the gender or ethnicity indicated by the name, and we see drastic differences in the response rates. I find it very hard to believe that any real good would come of making sure that everyone refereeing papers knew the identity of the paper's author.

[elisa freschi]

The problem of the good relations one needs to keep is, in my opinion, one of the causes of the lack of sincere book reviews (a topic to which I dedicated several blog-posts). I am afraid that if refereeing where no longer blind, it would also end up with a useless celebration of all projects written by "authorities" in the field. After all, we cannot expect from all scholars to be brave and honest at the same time. What would Moti's answer be?

[Moti Mizrahi]

Elisa: Thanks for your comment.

I am not sure that courage is required for being a good scholar. Intellectual honesty, on the other hand, is necessary. I may be naive but I still think that philosophers are in the business of seeking truth, not in the business of manufacturing BS. (I use BS in the technical sense here.)

As Lewis said, there are always people who try to "game" the system. As you said, some already do that (in the form of insincere book reviews). But perhaps lifting the veil of anonymity will make others call on such BS more often than not. Or maybe not. I don't know.