Our books






Become a Fan

« Job-market reporting thread (2018-19 season) | Main | Reader query on moving jobs »

08/14/2018

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Peter

Hi Marcus,

Thanks for the post. I don't mean to call into question whether what you suggest would be an improvement, but I do wonder how much of an improvement it would be on the specific issue of protecting authors. You mention that it is not uncommon for a less-than-well-established scholar to publish an idea, only for it to be credited to someone else (someone better established) who publishes something similar later on. If this happens now, I wonder whether the model you suggest will help much in terms of "scooping." Of course, someone could put the idea out first (and at least they could point to this), but if they are ignored and someone else is credited with the idea, are things much better than they were for the economists you discuss?

Marcus Arvan

Hi Peter: Thanks for your comment. I think those matters would be best addressed by combining an "open" peer-review system with new professional norms regarding scholarship. Further, I believe the relevant norms are likely to arise naturally as a result of transitioning to the more open peer-review model described. Allow me to explain.

I think one of main the reasons why "scooping" and wrongly crediting one person with an argument (rather than another person) has to do the fact that these very matters--namely, *who* gets credited with an idea--currently get settled through anonymized peer-review process. Here is why I think this matters: the incentive structures in anonymized review plausibly favor crediting Big Names with new ideas over Small Names or No-Names. Here's how.

In my experience, anybody who submits papers for anonymized review will tell you several things. First, the papers one chooses (and doesn't choose) to cite in a paper placed under review can very much be a way of "signalling" professionally. Generally speaking, reviewers will look for you to cite people *they* think are important--and, often enough, those people are Big Established names (since, all too often, people reviewing your paper are recognized names in the field themselves). So, what happens often enough in anonymized review is that authors are incentivized to cite and credit Big Names over Small Names or No-Names. I've heard some people say this pretty openly, in fact (that they choose to cite big names to "signal" to referees).

Consequently, I think the incentive structure of anonymized review generates the very problems this post alludes to (not to mention the fact that it incentivizes No-Names to hide their unpublished work but not so much Big Names).

On the other hand, It think the incentive structure with an "open" peer review system would be very different--tilted not in the direction of crediting Big Names but instead giving everyone a more equal opportunity to lobby for the *right* names to be credited, even before a paper is placed under review.

Why? Because in an open, public peer-review model literally anyone--Big Name, Middle, No-Name, or Outsider--can get in on the action, debating publicly (and anonymously, if need be) whether a new paper draft adequately credits the true originator of a given argument or idea. If anything would plausibly incentive proper credit-giving, I believe it is precisely this: leveling the field by giving the *profession at large* the opportunity to debate and correct for scholarly lapses before a paper is reviewed or published (rather than one or two reviewers).

I don't know about you, but if we moved to a more open peer-review model, I would very much be on the lookout for these sorts of things, advocating openly for corrections to improper scholarship--and I expect there are a good number of others who would be on the lookout as well. While some people might find that annoying--a kind of scholarly busybodiness--these are the kinds of things I think a healthy profession should have: a more public, transparent system of checks and balances to protect authors and ensure proper scholarship. And indeed, I would very much be thankful if such a system helped me improve *my* scholarship, helping me better cite and credit authors in the manner they are properly due.

Douglas W. Portmore

Hi Marcus,

As a graduate student coming out of a poorly-ranked PhD program, there was little incentive to read my unpublished work. But, due to our practice of peer-review, some really great philosophers (e.g., Derek Parfit and Stew Cohen) were "forced" to read my work either because they were an editor at a journal where I had submitted my work or because they had agreed to review it at the request of such an editor. The result was that I got a lot of valuable feedback on my work that I wouldn't have otherwise received. For these people wouldn't have read it but for this process being in place. And my graduate professors didn't provide the same quality or quantity of feedback that I received via the peer-review process. And no doubt part of it was that they had to interact with me face-to-face. As a result of this feedback, I was able to publish my work in some fairly prestigious journals and that led to people reading my work who wouldn't have otherwise bothered to do so. So, I wonder how would I end up receiving the same advantages on your proposed "open-online" peer-review model? Why would any great philosopher choose to read the paper of some graduate student at a poorly-ranked program? And why, even if they did, would such a person take the time to provide detail feedback on that work? If you've already addressed this issue in a previous post, you can feel free just to direct me there.

Also, in my experience as both an editor and a frequent reviewer, blind-review is indeed blind the vast majority of the time. There are times when I know who the author is and the editor wants me to review it anyway. And there are times that I have a hunch who the author is. But most of the time I don't know whether the person who wrote the article that I'm reviewing is a student or a full professor at an R1 university. And this ensures that keep on my toes and don't ever dismiss anything too quickly.

Marcus Arvan

Hi Doug: Thanks for weighing in! I think the anecdotes you offer are important in few respects.

First, they illustrate something many of us recognize--and that I certainly want to recognize: namely, that there are of course instances where our current peer-review system works well, benefiting our work and careers. Like you, I have had cases of wonderful reviewers and editors--people who took an unexpected interest in my work, read it charitably, helped me improve it immeasurably, and took a chance on publishing it!

However, I think the relevant question is not whether there are individual cases where everything works as it should, including cases where someone like Parfit takes an interest in your work as a result of our current review process (as an aside, I hear Parfit was a rare and exemplary person who routinely took interest and provided comments on work he was sent - which I think makes him admirable and heroic, but perhaps not representative enough to stake peer-review policies and practices on). The more relevant questions, I think, are which peer-review system--our current one or the more open one of math and physics--(1) operates better on average, and (2) which system makes the worse/more harmful errors when it does err.

In this post and many others (and in line with Heesen and Bright's empirical research), I've tried to argue that--anecdotes aside--there are systematic reasons to think that the open model is likely to work better on average and our current anonymized review system likely to make the worse and more harmful mistakes (as evidenced, I believe, in the current post).

Bearing that in mind, allow me to address the questions you raise: "So, I wonder how would I end up receiving the same advantages on your proposed "open-online" peer-review model? Why would any great philosopher choose to read the paper of some graduate student at a poorly-ranked program? And why, even if they did, would such a person take the time to provide detail feedback on that work?"

Here is my answer. I actually think it is more likely that a great philosopher like Parfit or Cohen would take an interest in your work in an "open" peer-review system than in our current one. This is for two reasons.

First, in our current system, whether your or my work ever ends up in front of someone like Parfit seems to me, in large part, a matter of luck. On the one hand, you first have to get lucky to submit your work to a journal with an editor like one of them (or, as in the case you describe, getting one of them assigned as a reviewer for a paper you submit). Then, in cases where they serve as an editor, whether they take a real interest in your paper may be a matter of how lucky you get with reviewers. If, for instance, you get unlucky with unsympathetic reviewers who pan the paper, then perhaps the editor involved may not give it the kind of close or sympathetic read they might have given it if different reviewers were involved.

In other words, while it is certainly *possible* for a submitted paper to end up in front of the right Big-Shot, and then get them interested in your work, it seems to me that a lot probably has to go right for that to happen.

As an analogy, I'd compare it to the way that talent scouts used to discover musical artists. In the old days, A&R people would troll popular music venues--but you had to get lucky: you had to be playing in the right club, on the right night, in front of the right A&R person, and have a great show. And if you were one of the many artists who got unlucky--who never lucked out in having the right talent scout in the audience who their music spoke to? Well, those are the artists you've never heard of.

The point here is that even if it's possible to "get lucky" in a particular system (as you may have with Parfit and Cohen), it's also possible to *not* get lucky. And so we have to once again ask which system is likely to perform better her on average.

So, then, how would you probably get noticed by people like Parfit and Cohen in the "open" system of math and physics? And why do I think there would plausibly be less luck involved?

The answer is that I have seen how things seem to work in math and physics. Things work like this: people post new work online. Work that is actually good then tends to get talked about openly: it is debated, shared by other people in the profession on social media, and so on. In other words, word of mouth builds about it--it spreads. At this point, if you are a Big Shot like Parfit or Cohen, you will probably be likely to hear about that work in one of two ways.

First, if you are on social media, you'll probably notice the chatter (this is what happens in math and physics)--and so you will then go check out the paper to see what all of the fuss is about. Second, if you are not online or on social media (as I hear some people in the profession are not), you will be likely to hear about it in the hallway, at conferences, or in a seminar. If, for instance, you are Parfit walking the halls of Oxford or Stew Cohen at Arizona, some grad student or other faculty member may be apt to say to say to you during a reading group, grad seminar, or personal discussion, "Have you read that new paper by Small Fry everyone is talking about on philpapers? You have to check it out. It's right up your alley."

All things considered, I think there is likely to be whole lot less luck involved here. To use a music analogy once again, talent scouts no longer just troll clubs looking for acts--as they recognize that there's too much luck involved in that (not to mention many scouts signing "can't miss" acts that totally miss!). No, what record labels do today (or so I hear) is just pay attention to which new artists get listeners and followers on Spotify, Tidal, etc. Why? Because they recognize that an open, crowdsourced method--paying attention to what people are actually listening to--is a better, more reliable way to find acts that are likely to sell lots of records.

My suggestion then is that the same likely applies in philosophy. You want to get someone like Parfit or Cohen to notice your work? It's *possible* in our current system of anonymized peer-review--but it's far more likely, I think (for reasons just given), in a open system where word gets around.

The same, I think, goes for feedback. As someone who follows physics, papers in physics get a great deal of online feedback--in part because that open system of peer review doesn't just involve one to three anonymous reviewers, but indeed anyone who happens to download and comment on the publicly available paper draft--including someone like Parfit or Cohen who would plausibly come across the paper by word of mouth.

Peter

Hi Marcus,

Thanks for the response. This seems plausible to me. I still have other worries about your proposed system (ones not directly related to this post, so I will leave them aside here), but I must admit, you are slowly winning me over.

Douglas W. Portmore

Hi Marcus,

Thanks for the detailed and thoughtful response. One remaining worry that I have is that the open system will not necessarily favor the best work but will almost certainly favor work that's shorter, less specialized, less technical, and otherwise more appealing to a wide audience. As your analogy suggests, "paying attention to which new artists get listeners and followers on Spotify, Tidal, etc....is a better, more reliable way to find acts that are likely to sell lots of records." But it is unlikely to be the best way to discover brilliant music that's not readily accessible to a broad audience. But, unlike record companies, the point of our peer-review system is not, I hope, to find the articles that will get the most "customers" but to find the articles that have the best insights.

Marcus Arvan

Hi Doug: Thanks!

Regarding your residual worry ("the open system will not necessarily favor the best work but will almost certainly favor work that's shorter, less specialized, less technical, and otherwise more appealing to a wide audience"), I think there are good reasons to think it is not likely to be realized.

First, it does not appear to be realized in math and physics (which use the open model). For example, perhaps the most famous papers in all of math right now are four-preprints by Shinichi Mochizuki that purport to solve the abc conjecture. https://en.wikipedia.org/wiki/Inter-universal_Teichm%C3%BCller_theory

These unpublished preprints total over 600 pages and purport to invent an entire new realm of mathematics that is so incredibly specialized, techincal, and novel that hardly anyone understands it. The papers have generated an incredible amount of interest and discussion among mathematicians for precisely these reasons--because they are experts *interested* in technical, specialized, sophisticated things, not merely math for mass consumption.

I think there is every reason to believe that things would work similarly in an "open" model of peer-review in philosophy--that is, that an open model will favor good work emerging and being discussed.

Sure, there would probably be some less technical papers that would gain a popular following (including interest among nonspecialists). But, first, I don't think that would be a bad thing, any more than it is a bad thing in physics (a discipline that has greatly benefited from popularization as well as interesting but ultimately flawed work by outsides, see e.g. https://en.wikipedia.org/wiki/An_Exceptionally_Simple_Theory_of_Everything ).

Second, and more to the point, I don't think there is any reason to think--given the interests of professional philosophers--that an open model would problematically favor shorter, less technical, philosophy "for mass consumption." For professional philosophy--in a quite obvious way--is not like Spotify. On spotify, the average listener is just your average person--someone with little expertise in music. In philosophy on the other hand (as in math and physics), the typical reader of philpapers pre-prints is going to be a specialist. And it is not as though philosophers interested in technical, specialized work--and able to discern good quality work from bad work--are going to suddenly disappear off the face of the earth or lose their interest in technically sophisticated work and ability to discern good work from bad work. No, in an open model philosophers would still be experts: there would still be the Derek Parfits, Stewart Cohens, Doug Portmores ;), and countless other serious professional philosophers who care about philosophical sophistication, quality, and so on--and would push online discussions of pre-prints in those directions, helping to recognize the best work out there.

Indeed, by my lights, the more open the system, the more likely this is to occur--as instead of leaving the judgment of a piece's quality up to a couple of anonymous reviewers (who can, as we see in the OP, get things very wrong), professional philosophers in the "open" system would be likely to do exactly what the mathematicians and physics do: debate the merits and deficiencies of new papers widely and openly--which I believe is likely to be at least as good a method for recognizing genuine philosophical merit (and probably better) than our current model.

Douglas W. Portmore

Thanks, Marcus. That's very helpful.

Another

Marcus,
I am inclined to agree with Douglas' first expressed view. I think our responses - and yours too - are largely a largely a function of our experiences. I feel I have had really fair treatment in the journal referee system. Of course there are particular papers with particular journals that may have gone not so well. But on the whole I feel despite the fact that for most of my career I was not at a high profile place, my papers were treated justly. Further, I have similar experiences to those Douglas reports. Senior important people in the field have said to me at conferences "I have read your paper ... it is great" ... or something of the sort. This has happened a number of times.

Marcus Arvan

Hi Another: Our responses may of course be influenced by our experiences. However, I do not think the arguments that I or others have given for changing peer-review ultimately boil down to personal experience.

As I have said, I have had some very good experiences in anonymized review, as well as some not-so-good ones. In this regard, from what I can tell I seem to quite typical. Like you and Doug, I have shared very positive experiences. However, I have also had some very negative experiences--and routinely encounter people reporting similar experiences on social media (such as capricious referees, absurdly long wait times, being deprived of credit for new ideas, etc.).

The arguments I have provided--and which Heesen and Bright provide--do not boil to mere anecdotes or experiences. As Heesen and Bright argue, there is literally *no* clear empirical evidence supporting the benefits of anonymized review, whereas there is quite a bit of evidence on the costs. Some of those costs are in turn exemplified clearly and disturbingly in the piece on economists I shared in this post. Finally, in the OP, other past posts, and my discussions in the comments section with Peter and Doug, I have tried to give detailed arguments for why various features and incentive-systems inherent in both models of peer-review favor the open model over our current one.

In short, while my background motivations for making these arguments may indeed have something to do with the experiences I have had personally and heard about second and third-hand--both good and bad, and in our discipline and other disciplines (math and physics)--I think the arguments themselves are *not* based on mere experience, but rather on a variety of converging lines of evidence: scientific evidence (of the sort Heesen and Bright point to), historical evidence (of the sort illustrated in the economics piece), and analysis of the features and incentive structures of rival models.

Like you and Doug, I have actually fared well in anonymized review. But that does not indicate to me that all is well, nor that we should content with the status quo given the overall balance of evidence. I advocate for change here because, when I survey all of the available evidence--and reason through it--the overall balance of evidence in favor of change seems to me clear (for many of the same reasons Heesen and Bright argue it is clear).

And Another

In Science as a Process, David Hull provides some interesting evidence about anonymous peer review in biology. Reviewing (many) referee reports for a journal on which he served on the editorial board, Hull found that it tended to be those who shared similar views as the authors of papers that were more critical in their referee reports. Hull then attributes this to the fact that scientists do not want papers in support of their view published that appear weak in argument. They thus are more critical of their allies.

Marcus Arvan

And Another: I haven't read Hull's piece, but judging from your description of it there seem to me several problems.

First, although it is an anecdote (and should be taken into account as such), it appears to a single editor engaging in post-hoc anecdotal without any control for potential bias (such as, potentially, wanting to find what he found). Unless I'm missing something (and please do let me know if I am, to be fair!), this is not an adequate design for detecting anything of statistical significance. Good empirical evidence normally requires multiple independent sources of evidence, control groups, etc.

Second, on that note, Heesen and Bright reference the empirical literature demonstrating little interrater consistency among reviewers, including one study of a top journal demonstrating that "“reviewers. . . agreed on the disposition of manuscripts at a rate barely exceeding what would be expected by chance.” (p. 18)

Chris Stephens

I've probably mentioned this before: as you probably know, philosophers of science have had access to the philosophy of science archive that Pittsburgh's HPS program hosts: http://philsci-archive.pitt.edu

But: it isn't set up for all the editorial and refereeing practices that you mention. However, it does have the advantage that if someone is worried about being scooped, they can post their paper there. It also makes the paper public in way that's a bit more than just putting it on your phil papers or home page. That would at least address some of the worries you raise.

I do worry that there are important disciplinary differences between philosophy and math/physics that might affect how well it would work for philosophers. It would be interesting to know if, say, history had set up and had a system similar to the one you propose, and to see how well it has worked for a more "humanities" discipline (Philosophy, of course, is really a hybrid discipline and only partially a humanities subject).


And another

Marcus
The study by Hull is more than anecdote, but I will leave it to you to find out.

Marcus Arvan

And another: Okay, will do!

Marcus Arvan

Peter: Thanks! :) I'd be curious to hear what your other worries are sometime.

Peter

Marcus,

I will leave my worries for another time (I want to go back and reread all of your other posts--my worries might be addressed already.)

For the moment, I have another question: let's suppose that you are correct. How do you think we might begin transitioning as a discipline to something like what you suggest? Do you think it might start best with some subdisciplines beginning something like this (I know philosophy of science already has something a bit like this)? Do you think this would work better than trying to get something together discipline-wide (perhaps through PhilArchive, for example)? It is an interesting problem since it involves both a change in behaviors and norms on the one hand and the creation of an infrastructure that is designed for this purpose. (It seems to me that PhilArchive is not quite suited to what you have in mind, at least not as it currently is, but maybe I am wrong.)

Marcus Arvan

Chris: Thanks for chiming in. Of course, we also have philpapers, which seems like it would be a natural fit for this function--though people don't use the way people use the pitt archive in philsci or the ArXiv in physics. Philpapers also doesn't currently have some important functionality that the ArXiv has, such as listing on a paper's page how many places it has been blogged about (along with links to discussions).

These are things I think we should think about (and philpapers should think about) moving forward. As I say in my reply to Peter below, I hope to write up a post on this stuff soon!

Marcus Arvan

Hi Peter: You sort of read my mind! I was just thinking this morning about writing up a post on this: on things we could do--as individuals and as a discipline--to transition in the direction of the "open online" peer-review model. I hope to add to this a few concrete things *I* am tentatively planning to do to promote such change.

As a brief preview, I think transitioning toward such a system will probably have to involve:

(1) People like me who favor and advocate for the system taking concrete steps that favor it, such as posting paper drafts online, blogging about unpublished drafts posted on the PhilArchive, and so on.

(2) People in positions of professional and institutional influence (e.g. the people who run Philpapers, etc.) taking steps to foster the new system, such as adding new functionality to philpapers and advocating for and acting according to new research norms (such as treating unpublished papers posted to the PhilArchive as the sort of thing one should be aware of, discuss, cite, etc.)

Anyway, this is just a preview of some possibilities. I hope to think things out a bit more as well as solicit ideas from the Cocoon's readers. But this is sort of what I am thinking: it's time to stop just talking about the "open" system but (at least for those of us who support it) taking active steps to realize it.

As a quick aside, I'm not inclined to think it would be best for specialty disciplines to do it. Philosophy of science has already done it (as you note), and it seems to have worked fine (in terms of people citing stuff posted to the philsci archive). But I think for it to *really* work it has to be more central to the discipline and centralized (as the ArXiv is in physics)--as I think it's only when we begin to take on new norms as *philosophers* (as opposed to just ethicists, etc.) that the new norms and practices are likely to stick. Which is another reason why I think philpapers/philarchive would be great. It's already probably the most widely trafficked site for philosophy research. By my lights, it should be our ArXiv, though much must be done to realize that promise!

Peter

Hi Marcus,

This might be a consideration for another post, but in your response to Chris you say "Philpapers also doesn't currently have some important functionality that the ArXiv has, such as listing on a paper's page how many places it has been blogged about (along with links to discussions)."

This makes me wonder whether math and/or physics has a different culture of blogging than philosophy does. Does a large percentage of academics in these fields blog about research? If so, is this a central reason for the success of the ArXiv system at this point in time? It seems possible to me to have a successful open system without a culture of blogging, but your comments make me wonder whether this particular open system is somehow intertwined with other aspects of academic culture.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)

Current Job-Market Discussion Thread

Job-market reporting thread

Writing Service


Categories