Why "anonymized" review isn't, and what to do about it

A very influential full Professor at a top-ranked program expressed concerns in my facebook feed about "anonymized" review that I've long shared: namely, that "anonymized" peer-review really isn't, and in ways that plausibly have elitist consequences.

What's the worry? The worry this philosopher expressed had to do with posting papers online and the way in which well-placed people get to present their papers at many conferences or move in circles where drafts are discussed. The concern, in brief, was this. It can be disadvantageous for an "unknown" person to post drafts of their papers online, but advantageous for a well-placed person at a ranked program to do so. Why? Among other things, because of "Google reviewing": the (unethical) practice of journal reviewers Googling paper titles, concepts, or phrases to figure out who the author is.

Indeed, I suspect I've had this happen to me many times myself, and know of many other people who have reported experiences just like my own: experiences of submitting papers to journals only to have my academia.edu analytics suspiciously reveal that the paper's title is Googled several times a few days or weeks later--this despite the fact that the paper hadn't been similarly Googled before submission (indicating, to me, at any rate, that someone--curiously, just after I submitted my paper--decided it would be a nice to idea to find out who wrote it!).

Although some reviewers may say they are "just curious" and that they can "be objective" despite Google reviewing, this is nonsense: we have biases, those biases are often insidious, and chances are, bias works against "unknown" people at low-ranked and in favor of well-networked people at top-ranked places. Indeed, a famous study in psychology, "Peer-review practices of psychological journals: The fate of published articles, submitted again", took articles by well-known authors that were already published in top-ranked journals, resubmitted them with fictitious names, and the vast majority of them were rejected, not because reviewers recognized they had already been published, but merely on account of their content (with reviewers often citing "serious methodological flaws"). Not only that, reviewers in other fields have been shown to be able to deduce authors' identities in anonymized review between 25% and 45% of the time.

And the problem in my view, is arguably much worse than this. The problem isn't merely Google reviewing. It is much more systemic. When I peruse my facebook feed, I see philosophers at top programs--including some early-career people who haven't published much, or at all--regularly moving in rarified circles: attending exclusive (sometimes invite-only) conferences or workshops, giving colloquium talks at other well-ranked programs, and rubbing elbows with some of the biggest names in the field. My noting these things might sound like envy--and maybe there is some latent envy there (who knows!)--but honestly, I wouldn't have a problem with it were it not for the ways in which I worry that it can corrupt anonymized review. For consider the following two people submitting their papers to a journal for "anonymized" review:

Person 1 works at a name program, and has presented their paper entitled, 'X', at numerous colloquia and conferences attended by influential people who specialize on the topic X is about. Many of these people know Person 1 personally, like them, and--if they were to receive a request to review X for a journal--would be likely to recognize the title of X or its general argument, thereby knowing (to a good approximation) that Person 1 wrote the paper. Not only that, let's say Person 1 (who, again, is at a name program) posts their paper online, so that other people who specialize on the topic know they wrote it and are from a top-program.

Person 2 works at a non-name program, hasn't had the opportunity to present their paper at conferences frequented by influential people on its topic, and either posts their paper online or doesn't.

Can we really expect "anonymized" review to function as it is supposedly intended--to ferret out and mitigate bias--in these cases? On the face of it, no. Given that Person 1 knows a lot of influential people--people who know they work at a highly ranked program--there are grounds for thinking that they may receive more favorable reviewers than Person 1: reviewers who know who they are, and are inclined (due to tacit, insidious biases) to judge the work of someone like them favorably. 

And things are worse than this still. Suppose Person 1 and Person 2 are working on similar papers, on similar topics, with similar arguments (I know many cases of this)--and indeed, that Person 2 has drafted a paper defending argument X first. Suppose Person 1, knowing they are in an advantageous professional position, posts their paper online, but Person 2, knowing they are not, does not. Person 1 may be more likely to receive favorable reviewers than Person 2, which might help them publish their paper first. Indeed, Person 2 may receive a line of unfavorable reviewers, causing their paper to bounce around at journals for several years. Worse still, since they did not post drafts of their paper online (to "preserve anonymized review", Person 2 does not even have an online record that they actually defended X first. In other words, even though Person 2 defended a similar argument as Person 1 before Person 1, Person 1 may be more liable to publish on it first, and thus, be recognized as argument X's originator, "scooping" Person 2.

Suddenly, "anonymized" review isn't looking so anonymized! Are there any better options? I've long suspected that they physicists have it right (see here and here). Anonymized review doesn't exist. It's a sham--and, given that it's a sham, the most equalizing thing to do is have something like the physicists' arxiv: a place where (1) everyone uploads drafts of their papers before sending them to journals, and (2) everyone is expected to cite papers that appear there, regardless of whether they have been published yet. In physics, in other words, journal publishing is sort of pro forma. It still happens, but the main action takes place on the arxiv: it's where everyone uploads drafts, and everyone is expected to cite papers that appear there.

Although this practice effectively does away with anonymized review--since everyone knows what everyone is working on--it nevertheless works (imperfectly, to be sure) as an equalizer. On the arxiv, whether you are a well-placed person or not, if your paper is out there, there is a professional expectation that your paper will be cited. If Person 2 comes up with argument X first, then, even if Person 1 has an advantage in the peer-review process, it is still expected that people cite Person 2, because their paper was on the arxiv first. The arxiv system is a way--an imperfect way, to be sure, but, I would say a far better way--to mitigate for biases than our present system. Philosophy should consider implementing an analogue. Philosophers of science already have an analogue--the PhilSci Archive--and, just like in physics, it seems (to me, at any rate) to work splendidly.

Or so say I. What say you?

Comments

[Sebastian Lutz]

Isn't PhilPapers already an analogue to the arXiv and the PhilSci Archive? I haven't used the PhilPapers upload feature so far, because I use the PhilSci Archive, but it seems to me to work in the same way.

[Marcus Arvan]

Hi Sebastian: Yes, but philpapers isn't used like the arxiv or PhilSci Archive. That's the difference. In physics, everyone posts their papers to the arxiv, and there is a disciplinary-wide convention/expectation to treat papers posted there as "published"--at least in the sense that people are expected to read and cite papers that appear there, even if they haven't been published in a journal yet. That's what makes it anti-elitist: namely, the social/professional conventions surrounding it. In contrast, (A) philosophers are not expected to post unpublished papers to philpapers (indeed, many people actively avoid posting unpublished papers there to "preserve anonymized review"), and (B) even if they do, if they aren't published yet no one cites them. That's why, although philpapers is nominally similar to the arxiv, it doesn't play the anti-enlitist role the arxiv plays. My suggestion is that, just as physicists treat journal publications as pro forms--citing arxiv papers *as* publications before they are accepted in journals--philosophers should do the same, and for the same reasons: because treating "anonymized" peer review is problematic.

[Elisa Freschi]

Marcus, I apologise for the naïve question, but how does it work in physics? I surmise that one does not just cite *whatever* is uploaded, even if it is just a delirous article by someone explaining physics on the basis of, say, Nostradamus. Is the access to arxiv restricted?

[Unnatural scientist]

Some clarification: PhilPapers does not posted unpublished work. So it is not like the arxiv in Physics.
There is another noteworthy difference. In many of the "hard sciences" it is quite clear when someone is making a contribution. Indeed, I referee for a science journal, and determining publish-ability is a rather straightforward matter, generally.
Philosophy is not at all like this. I have refereed 100+ papers for many journals, and some pretty unfinished and weak papers are sent in (OFTEN, TOO OFTEN). Would-be authors in philosophy seem less well situated to know when their papers are ready for public consumption.

[Elisa Freschi]

Academia.edu works also for unpublished articles (I myself frequently use for this purpose), but due to the reasons you mention, I cannot really imagine that one could have the epistemic duty to cite everything which is uploaded there.

[Marcus Arvan]

Hi Elisa: Thanks for your comment! A couple of thoughts.

The arxiv's editors do not allow authors to post articles that plainly fall short of any plausible level of "professional quality." They would presumably reject a delirious article on Nostradomus. So, there is some basic level of peer-review involved even there--though the standard is quite low.

Still, since the threshold is low, a ton of stuff does get published on the arxiv--and so your question (of what to cite) is a good one. As we have discussed on this blog before, one cannot cite everything. Still, physicists appear to err on the side of over-citing rather than underciting--and the fact that the arxiv sort of counts as "publishing" actually encourages people to write reply pieces to articles.

[Marcus Arvan]

Unnatural scientist: Since when does philpapers not post unpublished work?

I've posted unpublished work there before, and the FAQ portion of the site states that it "accepts works of all types...so long as they are of professional quality."

Also, the main difference you point to, "There is another noteworthy difference. In many of the "hard sciences" it is quite clear when someone is making a contribution", is simply not true for many areas of physics--particularly theoretical physics. There are tons of alternative systems of theoretical physics (string theory, twistor theory, etc.), and immense amount of disagreement over which approaches are promising or not. Further, people in physics actually engage with--rather than ignore--flawed approaches. For instance, a 2007 arxiv article by an unaffiliated Hawaiian surfer Garret Lisi, "An Exceptionally Simple Theory of Everything", has already received 100 citations, even though the article's main claims have been disproven. This, I believe, is a much bigger difference between philosophy and the hard sciences than the one you suggest: physicists engage with far more work--even flawed work--rather than ignoring it.

[David Wallace]

Two specific comments on arxiv:

1) Arxiv actually has a sponsoring system for publications. If you haven't published there before and you try to submit a paper, you won't be allowed; you have to have your paper vouched for by someone with several papers already on arxiv. ("Vouch for" doesn't mean "do the equivalent of peer review"; it means "check it's not silly").

2) I gather from colleagues in physics that it's not actually the case that people read things just because they're on arxiv: the volume has become too great. You read in your own narrowish area, but outside that, you tend to use things like journal acceptance and author name recognition as a filter. (It's still, though, correct that good scholarship requires citing arxiv papers if they're salient, whether or not those papers are published.)

And a general comment on arxiv: what often gets lost in these discussions (because they're mostly about the career effects of publishing) is that arxiv came about, and continues to exist, because it greatly helps research communication. The more that people can be encouraged to put work online (not at the "rough draft" level, but at the "ready for submission" level), the more quickly ideas can circulate and be engaged with.

[anon]

doesn't address the particular suggestion of emulating the physicists, and only part of the problem. But. Googling is preventable. Referees should have to sign something to the effect that they won't google the author. This simple reminder might be enough to get them to take the obligation seriously.

Also, those tempted to google might wish to note: many (most?) people who have a website also have google analytics or some other analytics program. If you google them and are a sort of obvious choice for refereeing their paper, they might surmise that you (unethically) googled their paper whilst 'anonymously' reviewing it. This has happened to me--I submitted a paper on topic X. Then someone from a university Y where there is a prominent expert on X googles my website... I don't *know* that they googled before making a decision on the paper. But that is sort of how I read it.

So, tempted googlers -- those you google *might well be able to guess that you unethically googled them.*