Some puzzles regarding anonymized review

A number of philosophers in my social media feeds have raised questions recently about anonymized journal reviewing: specifically, about cases where they suspect the identify of the paper's author, given the author's manner of citations--specifically, the manner in which it appears that the author may be citing themselves throughout their manuscript. The standard case of this is something like the following:

One gets asked to review a paper. The paper repeatedly cites a not-very-well-known author X, placing that author's previous work at the center of the paper under submission, developing X's previous work in new directions. Since X is not very well-known, it seems likely to the reviewer/potential-reviewer that X is the author of the current paper. 

A number of different people in my social media feeds have recently raised cases like this, asking social media friends what they should do. Should they decline to review? Should they alert the editor that anonymized review has potentially been compromised, and let the editor decide? Etc. Although there have been disagreements in my feeds about what the right answer to these questions are, the most popular answers seem to be that reviewers in this kind of situation should either go ahead with the review if they are not certain who the author is, or alternatively, let the journal editor[s] know of the situation and let the editor decide. These both seem like reasonable answers to me, especially the latter--as reviewers are tasked with reviewing manuscripts by the journal editors [who therefore, it seems to me, should be consulted if there is any doubt].

However, I'd like to step back from these questions to some broader questions about how anonymized review can be compromised, and what ought to be done to "protect" anonymized review. As readers will see, I think there are some very serious puzzles to grapple with here that seem to me inadequately addressed at present--and indeed, I am not sure that anonymized review can be protected in a way that is either [A] effective, and/or [B] fair to authors. Let me explain.

I think perhaps the best way to introduce the host of puzzles I think arise is to reflect a bit more on the case I began this post with: the case where reviewers suspect they know an author's identity due to patterns of "apparent self-citation." To begin, it seems to me clearly legitimate for authors to write and attempt to publish papers further developing work they have previously published. If author X has published work defending an original argument or theory, they have convinced peer-reviewers in the past that the argument or theory in question is worth presenting to specialists for consideration. Further, even if that argument or theory has not had much "uptake" in the literature--for instance, if it has been largely ignored or perhaps cited only a few times in the literature--that does not seem to me sufficient reason to think the author should not be able to publish work further developing that argument or theory, as a large part of why they might do so might be to further demonstrate and develop its importance [in part due to demonstrating the argument/theory's fecundity, but also perhaps due to publishing the latter work in more highly-ranked journals than their earlier work]. To take a concrete case, even if Einstein's special theory of relativity didn't have much uptake in months or years after its 1905 publication in Annalen der Physik, it seems to me absurd to suppose that Einstein shouldn't be able to, say, submit his work defending the General Theory unless and until the former theory had serious uptake. The uptake, or lack thereof, of a person's work in a given area can have all kinds of explanations, including bias against the author or their ideas [as we all know, many controversial ideas in scientific and intellectual history were profoundly unpopular in the immediate aftermath of their publication--including, for example, Hume's Enquiry, which Hume said, "fell dead-born from the press"].

Now consider how an author in this position--an author attempting to build on their previous work--might attempt to "ensure" anonymized review. Here, there are roughly two options available:

1. Cite their previous work, but in the third person [for instance, in my case, Arvan XXXX].
2. Redact citations of their previous work, as in "[Author redacted to preserve anonymized review]".

On my social media feeds, some people have argued for the superiority of option [2], noting that some journals apparently require it in cases like these. But, is option [2] really superior? Does it "preserve anonymized review"? The answer, it seems to me, is plainly no, at least not in the kinds of cases described above: the case of a not-very-well-known author. To see how, suppose the author in question, Author X, is the only person to defend a given argument or theory in the literature. If the current paper under submission develops that argument or theory further, then "redacting" citations of the person's previous work actually accomplishes the opposite of redaction: it signals to readers precisely who the author is--as, by inference, the redacted person's identity could only be the person who previously published that argument or theory. In contrast, approach [1], while it may suggest that X is "probably" the author, at least leaves it an open possibility that they aren't the author [it could well be possible--if only unlikely--that someone else is systematically citing X's work].

Anyway, here's the problem: it doesn't seem like either [1] or [2] really solves the problem of "protecting" anonymized review. For, although some of us might like to assume the best about reviewers--namely, that they would not infer an author's identity from the above information--there is evidence that some reviewers, at least, aim to discern authors' identities when reviewing their papers, going so far as to "Google review" paper titles. Consequently, it seems as though there is really no very good way to "preserve anonymized" review in cases like these. If the reviewer reviews the paper, either on proposal [1] or [2], they can have a very good idea of who the paper's author likely is--and indeed, it is highly likely that any reviewer of the paper will be in a similar position, as in cases like [1] or [2] any reasonably intelligent person can infer who the author likely is. Conversely, if reviewers routinely decline to review papers in cases like these, authors are in effect "punished" for attempting to build on their previous work. Long story short: in cases like these, true "anonymized review" is nearly-impossible. Either no one reviews the paper, which is problematic, or someone does, which is also problematic, as pretty much anyone who reviews it will have a good idea of who the author probably is. 

Now, you might say, this is a unique case: most cases aren't like it, as the typical journal submission involves authors citing other people's work. However, this is far too quick, as there are countless other ways in which "anonymized" review is routinely jeopardized--among them:

1. People archiving paper drafts on their webpage[s] and/or announcing it on their social media feed: In my experience, this is quite common. Many people upload drafts of unpublished papers online. In some cases, people go so far as to announce it on social media, noting to large numbers of other people in the field, "I just uploaded paper X to my website-check it out! 
2. People presenting paper drafts at numerous conferences and department colloquia, including invite-only conferences with many of the most notable specialists in their field: These practices not only expose potential reviewers to papers directly, but can also do so indirectly, though relayed conversation [viz. "So-and-so has an interesting paper on that topic. I saw it presented at conference C"].
3. People [particularly job-candidates] referencing their research programs on their website: Even if one does not archive actual paper drafts on one's website, it is common practice for job-candidates to post research statements and CVs, both of which can give visitors [potentially, anyone who searches a key phrase in a paper] a pretty good idea of whether they are the author of an "anonymized" paper under review. 
4. PhD Dissertations are routinely archived online by Universities: Mine was, and anyone searching a key phrase from my early work [e.g. "nonideal original position"] could identify me as the likely author of many of my early papers.

In other words, in the modern internet-age, the ways in which anonymized review can be compromised are numerous, and increasingly easy to compromise. There are, of course, all kinds of ways in which some of these potential compromises might be "addressed", such as by journals prohibiting self-archiving drafts online prior to publication [on at least one reading, Phil Review's submission guidelines imply this]. But this at most "solves" one of the above problems--the problem of self-archiving drafts--and not, I think, in an adequate manner. First, there are perfectly legitimate scholarly reasons to self-archive online: to get feedback on unpublished work, etc. Second, in the history of intellectual thought, there have been many priority disputes--disputes over who arrived at an important scientific or philosophical result first--and self-archiving online is one way to establish priority. Although I know people who think priority disputes are tasteless--that it really shouldn't matter who "arrived at an idea first"--this is not only contentious [priority disputes do occur, and their being improperly resolved can be plausibly unfair to people whose work is not recognized, such as Rosalind Franklin]. Third, and most importantly, not allowing self-archiving not only does nothing to address [b]-[d]; it actually amplifies [b], as self-archiving online is one way that not very-well-placed people can even the playing field with people who are better placed [e.g. invited to more conferences, etc.].

Finally, I'm not sure what can really be done about [b]-[d]. What are we going to do: address [b] by telling people they cannot present at conferences and colloquia [for fear of compromising anonymized review]? Address [c] by tell people they shouldn't post research-statements or CVs online? Etc. In the modern internet age, these "solutions" seem increasingly absurd. The internet itself appears--to me at rate--to systematically compromise "anonymized" review. And so, it seems to me, we need to rethink "anonymized" review itself. As David Wallace points out in the comments section here and I have pointed out before here, academic mathematics and physics moved away from "anomymized" review long ago for more or less these reasons, and indeed, to improve academic communication [viz. the longer a paper is kept out of the public sphere, the longer its ideas are withheld from other researchers]. The alternative approach in math and physics today is to archive unpublished drafts on a central public site, the ArXiv, and see which ones have uptake in discussion. Paper drafts--both by "famous" and "non-famous" authors--are routinely posted, discussed, and critiqued prior to publication by experts in the field. Indeed, this public peer-review is actually arguably superior to preventing bias [including status quo and prestige bias], as online discussions are held both by famous and non-famous people--so, the strength and publishability [or lack thereof] of a given paper emerges publicly, as opposed to being decided by one, two, or three journal reviewers who may have kinds of all kinds of biases themselves. As David Wallace explains, and I concur, the workability--and advantages--of this kind of public vetting are pretty clear. It is not a perfect system, by any means [biases can of course still affect how new papers are received. But at least it is open and broadly "democratic"--enabling everyone to judge the quality of unpublished papers, as opposed to a couple of "anonymous" reviewers. And, as we have seen above--and, I think, we all know--"anonymized" review isn't perfect either, not by a long shot. "Anonymized" review is not only arguably routinely compromised; it also plausibly disadvantages non-well-placed scholars, plausibly generates priority disputes, and [as Wallace emphasizes] significantly hampers the efficient dissemination of research.

So, then, is it time to rethink "anonymous" review in philosophy? I think so--though, as always, I'm willing to rethink my view. What do you all think?

Comments

[Nick Byrd]

I'm actually writing a post about the possibility of doing away with compulsory peer review entirely (it'll go up Sunday or Monday). As Kate Norlock has said, eliminating compulsory peer review altogether seems right to me on grumpy days. But this post isn't grumpy. I consider potential costs and benefits and unknowns. Spoiler: I don't think that eliminating compulsory peer review would be obviously worse than existing peer review practices.

Thanks for continuing this discussion!

[Referee]

Marcus,
I have reviewed 100+ papers for journals. At least half of them for very good journals (Mind, Phil Q, Phil Sci, Australasian JP, Canadian JP, BJPS, Synthese). I would guess that about 5-20 of the 100 or so papers I have reviewed were ultimately printed in the journals. But others went on to be published in other weaker journals (more or less unchanged, unfortunately!). In most cases (I would say 95%) I had no idea whose paper I was referring. I discover whose paper it is when it is published. When I think it is obvious whose paper it is, and I think this may lead to a conflict of interest, I recuse myself.
So I think anonymous peer review does work (that is, it keeps things anonymous).
I do not think the community of scholars working on any particular issue is so small that we know all the players (or potential players).

[Marcus Arvan]

Hi Nick: Thanks for chiming in, and for your kind words. I look forward to reading your upcoming post, and please do let me know when it posts - I would be happy to link to it!

[Marcus Arvan]

Hi Referee: I applaud your integrity and conscientiousness. However, I think one must be careful generalizing from one's own case to the conclusion that "anonymous peer review does work."

First, to clarify, my post does not suggest that peer review does not work simpliciter. My experience is that it does "work", but only very imperfectly--and my aims in this post are merely to [i] draw attention to some of its imperfections, [ii] explain why I think are amplified in the modern internet age, [iii] explain why I do not think they have very good solutions within existing practices, and [iv] once again draw attention to another model which I think plausibly works far better in the internet age].

Second, although you may be very conscientious, do not engage in problematic reviewer behavior [e.g. Google reviewing], and thus do not have any idea who wrote a given paper 95% of the time, there are several related reasons to believe that not all reviewers are like you. First, at least one journal editor has said that people tell him they Google review [http://philosopherscocoon.typepad.com/blog/2013/02/not-cool.html ]. Second, I have pretty compelling evidence that I have been Google reviewed on multiple occasions. The analytics for my professional webpage and academia webpage both track "search terms" that draw people to my page. On a number of different instances with different papers at different journals, my analytics reported that someone searched the title of the paper under review either just after the journal's manuscript system changed to "under review/sent to reviewers", or just before I received a decision email from an editor. In some cases, this was especially suspicious given that no one had arrived at my page searching that paper's title at any time several months prior or after my paper being placed under review. Although this of course does not prove that I was Google reviewed, it is very suggestive--and I have heard many, many people [on my social media and elsewhere] report experiencing the same thing multiple times as well.

Third, although communities of scholars working in different areas are not so small that we know all the players, my experience as a reviewer is that I am often asked to review papers on *very* specific issues within my subfield. For instance, I write on Rawlsian nonideal theory--something that, when I started in the area, *very* few people had written on. I am often asked to review papers on this very narrow issue, and know that some of my papers were reviewed by a very narrow number of people working in the same area. On one occasion, one of the main people in my area left their name in the properties section of their review. On another occasion, I learned that another person in the same very small group of scholars--who I had previously met at a conference where I presented the paper--reviewed my paper for another journal. So, it seems to me, reviewers may be selected in a manner that, by trying to "match" reviewers to topics, dramatically narrows the likely class of reviewers to a relatively small class of people, many of whom are likely to know each other, or at least each other's work--particularly if they present regularly at conferences, department colloquia, etc. Further, the concern here seems to me potentially amplified when it comes to highly ranked journals [viz. who probably reviews many papers on metaphysical grounding submitted to Nous or PPR? I think one could probably guess likelihoods pretty well, given the journal's standards and the people who are considered serious players in the ground literature].

Finally, there are empirical reasons to believe that peer-review does not track article quality nearly as reliably as many of us would like to believe . For instance, one recent study in psychology found that when a number of articles published in top journals in the field were *resubmitted* to the same journals, only 8% of reviewers detected the resubmissions, and "Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws"." [http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=6577844 ]. In other words, 89% of the time, articles that were once accepted into top journals in the field were subsequently regarded as not worthy of publication in those very same journals. Although the study admittedly concerned non-anonymized review procedures, it illustrates--in my view--just how noisy judgments of research quality can be. We tend to think we can reliably detect article quality--but there is, to my knowledge, no clear empirical verification that we are anywhere as reliable as we may like to suppose. This, in my view, is another reason to favor the "open access" approach to publication utilized in math and physics [where large numbers of researchers are publicly exposed to, read, and critique paper drafts posted to the ArXiv. The cream, as they say, tends to rise to the top--and, given the Condorcet theorem, I am more apt to trust the judgments of the many than the judgments of a few].

[Wesley Buckwalter]

I have begun omitting names and/or suppressing references during self-citation, similar to your option 2. After that, I am not responsible for the inferences of others.

So many broad threats to anonymity in the internet age, together with doubts about whether it is often enough actually achieved in practice leads one to wonder whether anonymous review should be replaced with a more transparent reviewing procedure.

[B.M.]

Would a third option be to delete rather than redact self-citations? If the paper is accepted, self-citations could then be added to the final version prior to the proofs stage (I don't know how most journals would feel about that). If the author's work is not well known, presumably most referees wouldn't notice that certain papers that ought to be cited are not being cited.

[Marcus Arvan]

Hi B.M.: Thanks for the suggestion. Although that might be feasible in some cases, it is infeasible in the kind of case I am most concerned with. The case I am most concerned with is not one where the author merely cites their own work on occasion, but where the entire central line of argument builds significantly on past work they have published--where, for instance, they summarize in detail some of their past arguments, and either go onto develop further arguments [taking the past ones as background], or applying a theory they have defended to some new area [for instance, extending a moral theory they have defended to a debate in applied ethics]. In cases like this, it is impossible to summarize one's past work without--in some way or another--signalling what work it is [and presumably, since it is being examined in great detail, should be cited extensively].

[Sam Duncan]

One related case to the one Marcus mentions is one where a paper seem to draw heavily on the research of an established, though possibly not super famous figure, without citing that person. In those cases I can see it possibly making a difference who the person is. I had a case like that once but it was so unoriginal that it really didn't matter if it was the person in question pouring new wine into old bottles or someone else copying them without attribution, but I can see more marginal ones where the paper might be seen to make a contribution if it were the person question expanding on their work, but might not if it were just someone else copying them (and at the very least pretending one's work is more original than it is may merit a rejection depending on the case). What should one do in a case like that? Would google reviewing be the ethical thing? I'm just curious what people think.