Our books






Become a Fan

« Saturday Sounds | Main | How do you get things done? »

06/13/2016

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Nick Byrd

I'm actually writing a post about the possibility of doing away with compulsory peer review entirely (it'll go up Sunday or Monday). As Kate Norlock has said, eliminating compulsory peer review altogether seems right to me on grumpy days. But this post isn't grumpy. I consider potential costs and benefits and unknowns. Spoiler: I don't think that eliminating compulsory peer review would be obviously worse than existing peer review practices.

Thanks for continuing this discussion!

Referee

Marcus,
I have reviewed 100+ papers for journals. At least half of them for very good journals (Mind, Phil Q, Phil Sci, Australasian JP, Canadian JP, BJPS, Synthese). I would guess that about 5-20 of the 100 or so papers I have reviewed were ultimately printed in the journals. But others went on to be published in other weaker journals (more or less unchanged, unfortunately!). In most cases (I would say 95%) I had no idea whose paper I was referring. I discover whose paper it is when it is published. When I think it is obvious whose paper it is, and I think this may lead to a conflict of interest, I recuse myself.
So I think anonymous peer review does work (that is, it keeps things anonymous).
I do not think the community of scholars working on any particular issue is so small that we know all the players (or potential players).

Marcus Arvan

Hi Nick: Thanks for chiming in, and for your kind words. I look forward to reading your upcoming post, and please do let me know when it posts - I would be happy to link to it!

Marcus Arvan

Hi Referee: I applaud your integrity and conscientiousness. However, I think one must be careful generalizing from one's own case to the conclusion that "anonymous peer review does work."

First, to clarify, my post does not suggest that peer review does not work simpliciter. My experience is that it does "work", but only very imperfectly--and my aims in this post are merely to [i] draw attention to some of its imperfections, [ii] explain why I think are amplified in the modern internet age, [iii] explain why I do not think they have very good solutions within existing practices, and [iv] once again draw attention to another model which I think plausibly works far better in the internet age].

Second, although you may be very conscientious, do not engage in problematic reviewer behavior [e.g. Google reviewing], and thus do not have any idea who wrote a given paper 95% of the time, there are several related reasons to believe that not all reviewers are like you. First, at least one journal editor has said that people tell him they Google review [http://philosopherscocoon.typepad.com/blog/2013/02/not-cool.html ]. Second, I have pretty compelling evidence that I have been Google reviewed on multiple occasions. The analytics for my professional webpage and academia webpage both track "search terms" that draw people to my page. On a number of different instances with different papers at different journals, my analytics reported that someone searched the title of the paper under review either just after the journal's manuscript system changed to "under review/sent to reviewers", or just before I received a decision email from an editor. In some cases, this was especially suspicious given that no one had arrived at my page searching that paper's title at any time several months prior or after my paper being placed under review. Although this of course does not prove that I was Google reviewed, it is very suggestive--and I have heard many, many people [on my social media and elsewhere] report experiencing the same thing multiple times as well.

Third, although communities of scholars working in different areas are not so small that we know all the players, my experience as a reviewer is that I am often asked to review papers on *very* specific issues within my subfield. For instance, I write on Rawlsian nonideal theory--something that, when I started in the area, *very* few people had written on. I am often asked to review papers on this very narrow issue, and know that some of my papers were reviewed by a very narrow number of people working in the same area. On one occasion, one of the main people in my area left their name in the properties section of their review. On another occasion, I learned that another person in the same very small group of scholars--who I had previously met at a conference where I presented the paper--reviewed my paper for another journal. So, it seems to me, reviewers may be selected in a manner that, by trying to "match" reviewers to topics, dramatically narrows the likely class of reviewers to a relatively small class of people, many of whom are likely to know each other, or at least each other's work--particularly if they present regularly at conferences, department colloquia, etc. Further, the concern here seems to me potentially amplified when it comes to highly ranked journals [viz. who probably reviews many papers on metaphysical grounding submitted to Nous or PPR? I think one could probably guess likelihoods pretty well, given the journal's standards and the people who are considered serious players in the ground literature].

Finally, there are empirical reasons to believe that peer-review does not track article quality nearly as reliably as many of us would like to believe . For instance, one recent study in psychology found that when a number of articles published in top journals in the field were *resubmitted* to the same journals, only 8% of reviewers detected the resubmissions, and "Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws"." [http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=6577844 ]. In other words, 89% of the time, articles that were once accepted into top journals in the field were subsequently regarded as not worthy of publication in those very same journals. Although the study admittedly concerned non-anonymized review procedures, it illustrates--in my view--just how noisy judgments of research quality can be. We tend to think we can reliably detect article quality--but there is, to my knowledge, no clear empirical verification that we are anywhere as reliable as we may like to suppose. This, in my view, is another reason to favor the "open access" approach to publication utilized in math and physics [where large numbers of researchers are publicly exposed to, read, and critique paper drafts posted to the ArXiv. The cream, as they say, tends to rise to the top--and, given the Condorcet theorem, I am more apt to trust the judgments of the many than the judgments of a few].

Wesley Buckwalter

I have begun omitting names and/or suppressing references during self-citation, similar to your option 2. After that, I am not responsible for the inferences of others.

So many broad threats to anonymity in the internet age, together with doubts about whether it is often enough actually achieved in practice leads one to wonder whether anonymous review should be replaced with a more transparent reviewing procedure.

B.M.

Would a third option be to delete rather than redact self-citations? If the paper is accepted, self-citations could then be added to the final version prior to the proofs stage (I don't know how most journals would feel about that). If the author's work is not well known, presumably most referees wouldn't notice that certain papers that ought to be cited are not being cited.

Marcus Arvan

Hi B.M.: Thanks for the suggestion. Although that might be feasible in some cases, it is infeasible in the kind of case I am most concerned with. The case I am most concerned with is not one where the author merely cites their own work on occasion, but where the entire central line of argument builds significantly on past work they have published--where, for instance, they summarize in detail some of their past arguments, and either go onto develop further arguments [taking the past ones as background], or applying a theory they have defended to some new area [for instance, extending a moral theory they have defended to a debate in applied ethics]. In cases like this, it is impossible to summarize one's past work without--in some way or another--signalling what work it is [and presumably, since it is being examined in great detail, should be cited extensively].

Sam Duncan

One related case to the one Marcus mentions is one where a paper seem to draw heavily on the research of an established, though possibly not super famous figure, without citing that person. In those cases I can see it possibly making a difference who the person is. I had a case like that once but it was so unoriginal that it really didn't matter if it was the person in question pouring new wine into old bottles or someone else copying them without attribution, but I can see more marginal ones where the paper might be seen to make a contribution if it were the person question expanding on their work, but might not if it were just someone else copying them (and at the very least pretending one's work is more original than it is may merit a rejection depending on the case). What should one do in a case like that? Would google reviewing be the ethical thing? I'm just curious what people think.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)

Subscribe to the Cocoon

Job-market reporting thread

Current Job-Market Discussion Thread

Philosophers in Industry Directory

Categories

Subscribe to the Cocoon