Protecting authors in peer-review

Readers may recall that I have dedicated quite a bit of time to arguing for systemic changes to our peer-review process--specifically, for moving away from "anonymized review" in favor of the kind of "open online" peer-review model utilized in math and physics. Why I have I dedicated so much attention to this topic?

One set of reasons (the ones emphasized in the posts linked to above) is that I think such changes are likely to have substantial benefits for everyone--for authors, reviewers, editors, and the profession at large (by, among other things, improving the dissemination of new research). I have tried to draw attention to these benefits in large part because, or so I have argued, the oft-ascribed benefits of "anonymized" review are dubious and probably illusory ("anonymized" review really isn't in the digital age). Interestingly, Remco Heesen (Cambridge) shared a draft of a paper co-authored with Liam Bright (LSE) defending similar conclusions using empirical research:

We think there are a number of clear benefits to abolishing pre-publication peer review. In contrast, while various benefits of the existing system (downsides of abolishing peer review) have been suggested, we do not think there exist any that have clear empirical support. Insofar as empirical research exists, it is ambiguous in some cases, and speaks relatively clearly against the claimed benefit of the existing system in others...[Consequently], we claim that the present state of the evidence suggests that abolishing pre-publication peer review would lead to a Pareto-improvement: each factor considered is either neutral or favors our proposal. (p. 3, my emphasis)

However, the likely benefits alluded to above--to authors, editors, reviewers, and the profession--are not the only reasons that I think we should favor change. More deeply, I have long worried that "anonymized" review--far from ensuring fairness (as often claimed)--plausibly places authors at serious and undue career risks, particularly early-career and less-established authors. Allow me to explain.

In the comments section of my recent post on brutal reviews and rejection letters of famous works, Ingrid Robeyns (Utrecht) shared a remarkable paper, "How Are the Mighty Fallen: Rejected Classic Articles by Leading Economists." In my view, anyone interesting in defending "anonymized" review should read it, as I think (following Heesen and Bright's paper) it pretty thoroughly illustrates how illusory the supposed benefits of the peer-review model probably are (including how, even pre-digital age, "anonymized" review often wasn't, see pp. 172-3 in particular). However, by my lights perhaps the most striking thing about the paper is the ways it illustrates how "anonymized" review may routinely lead authors to face serious harms. Here are a few cases:

Early in his career, [Gary Becker] submitted what became "Competition and Democracy" to the Journal of Political Economy. "The then editor Earl Hamilton agreed to publish it. He eventually withdrew the commitment because of negative comments by Frank Knight, who was one of the people who refereed the paper. I still have a copy of Knight's referee report, and I cannot say that I am any more impressed by it now than at that time." Becker "became discouraged by the report and put the article away" until he finally published it in the Journal of Law and Economics several years later. However, by that time, other articles had been published that employed the same approach. (p. 169; my emphasis)

[L]ike Becker, Krugman notes that, even if another journal eventually prints a paper, the delay that initial rejection causes may permit others to beat the paper into the intellectual market. Krugman sent his "Target Zones and Exchange Rate Dynamics" to the Journal of Political Economy. "This time I got two favorable referee reports. The paper was nonetheless rejected ... by [the referee] who thought that the paper was of 'insufficient general interest' for the JPE. The paper didn't come out (in the QJE) until August 1991. By that time the target zone literature, all of which made use of the techniques first introduced in my paper, had exploded, and consisted of at least a hundred published and unpublished pieces; in fact, I had to add a postscript to the QJE version referring to subsequent literature." (p. 170)

"Only after years of rejections by four journals did Brian Arthur's "Competing Technologies, Increasing Returns, and Lock-In by Historical Events" appear in the Economic Journal...The paper began a six-year odyssey. "First it was dismissed at AER in desultory fashion. Then I submitted it to QJE, and it was turned down there. Then because Clower had left AER I resubmitted it to AER. It underwent one refereeing go-round, followed by two appeals. Finally, two years after this second submission, AER turned it down again. ... I then submitted it to EJ; and it got turned down. I appealed; and finally, in 1989, EJ published it...Like other authors, Arthur suggests that delay from the rejections threatened his ideas' currency. "Because papers based on mine had started to appear in the literature," referees told him that "the idea ... is already recognized in the literature." (p. 173)

"Keynes rejected Roy Harrod's article that first sketched the marginal revenue curve. Although the Economic Journal finally published the article years later, Harrod felt that the delay in publication cost him credit for the new concept...Paul Samuelson remembers, "Roy Harrod went to his grave bitter because Maynard Keynes, absolute monarch at the Economic Journal, turned down his early breakthroughs in the economics of imperfect competition. Thus, Harrod was robbed of credit for the 'marginal revenue' nomenclature. All this was on the advice of Frank Ramsey, genius in logic and mathematics. " (p. 175)

To some readers, these kinds of concerns--about authors being deprived of credit or 'priority' for a scholarly idea--may seem petty. Shouldn't "the point" of scholarship not be so much a matter of "who gets the credit" but rather the ideas themselves, as it were? What does it matter who gets the credit for a given idea? These are reactions I have encountered fairly often. But let me try to explain--using a few real life stories I have heard before--why I think these reactions are normatively mistaken.

A few times over the years, I have heard variants of the following story. The story involves a graduate student or early career person working with a less-than-scrupulous (or perhaps unintentionally obtuse) well-established member of the profession--a faculty mentor or dissertation advisor, for instance. The early-career works tirelessly on developing an idea for publication, yet has difficulty publishing it. Then, a few years down the road, the more well-established person they worked with publishes something eerily similar, 'scooping' them as it were. And unfortunately, because the person is well-established (and may be in a position of professional power), the vulnerable person whose idea was plausibly 'scooped' has little recourse. They may feel powerless to even raise the notion that a professional and scholarly wrong might have been done. While I have no idea how common these types of cases are, I've heard of at least a few of them--and they seem to me the sort of thing that professional institutions and norms should guard against.

Here's another kind of case I have encountered seemingly much more often (and indeed, I have heard others remark on this too). A less-than-well-established scholar publishes Argument X in the literature, perhaps in a lower-ranking journal. Then, a few years later, a much more established scholar publishes a similar argument, perhaps in a more highly-regarded journal--without referencing the less-well-established scholar's work (perhaps entirely innocently, not knowing that work exists). Alas, in the subsequent literature, the well-established figure becomes associated and credited with X, and the less-established author's contribution ignored or suppressed (I have at least one now-well-established acquaintance in the profession who has mentioned suffering this fate). This too, I think, is the kind of thing professional institutions and norms should guard against.

One additional reason why these things should be guarded against, I think, is the anecdotal sense (though it is just that) these kinds of cases may have a tendency to disproportionately affect vulnerable or otherwise-marginalized members of the profession--individuals who, for one reason or another, feel or are powerless to rectify whatever wrong(s) they think may have occurred (and again, it is worth emphasizing that some or even many such wrong(s) may be completely inadvertent, such as someone genuinely not remembering that it was a student of theirs who initially thought of X several years ago, or someone not aware that Argument X existed in the literature because they simply missed the article where it originally appeared).

In any case, the points I would like to make are these. First, I do not think there is anything petty in worrying about these cases. Whether the publication of an idea is delayed for years by 'anonymized' review, such that a person is (either inadvertently or intentionally) 'scooped' or otherwise denied credit for a contribution: these things can have serious and lasting negative effects on a person's career (such as whether they get a job, get tenure, etc.). Second, for reasons noted above, it is plausibly the more vulnerable members of the profession who, due to their positions of professional vulnerability, probably disproportionately face these risks and harms. Finally--and perhaps most importantly--these are yet another set of problems that the "open" peer-review model in math and physics would plausibly address. For, as I have mentioned before, in that model (1) researchers post working drafts of articles on an online repository before sending them to journals, (2) people are expected to be aware of, discuss, and cite those articles (even before acceptance by a journal), and (3) public discussion prior can catch and correct scholarly errors (including matters relating to priority and appropriate credit in citations) before new works are placed under review or published. In the math and physics model, if a graduate student uploaded a paper on Argument X years ago, it would be possible and (judging from how things work in math and physics) incentivized for people to help ensure that work is appropriately credited.

These are, I believe, very important plausible benefits to be aware of. In addition to the other plausible benefits of an open peer-review model (and lack of adequate empirical support for the benefits of "anonymized" review), one thing we should want out of a publication system is to protect authors, ensuring they are appropriately recognized in the scholarly record--particularly when it is more vulnerable, less-established authors whose careers most lie in the balance.

Comments

[Peter]

Hi Marcus,

Thanks for the post. I don't mean to call into question whether what you suggest would be an improvement, but I do wonder how much of an improvement it would be on the specific issue of protecting authors. You mention that it is not uncommon for a less-than-well-established scholar to publish an idea, only for it to be credited to someone else (someone better established) who publishes something similar later on. If this happens now, I wonder whether the model you suggest will help much in terms of "scooping." Of course, someone could put the idea out first (and at least they could point to this), but if they are ignored and someone else is credited with the idea, are things much better than they were for the economists you discuss?

[Marcus Arvan]

Hi Peter: Thanks for your comment. I think those matters would be best addressed by combining an "open" peer-review system with new professional norms regarding scholarship. Further, I believe the relevant norms are likely to arise naturally as a result of transitioning to the more open peer-review model described. Allow me to explain.

I think one of main the reasons why "scooping" and wrongly crediting one person with an argument (rather than another person) has to do the fact that these very matters--namely, *who* gets credited with an idea--currently get settled through anonymized peer-review process. Here is why I think this matters: the incentive structures in anonymized review plausibly favor crediting Big Names with new ideas over Small Names or No-Names. Here's how.

In my experience, anybody who submits papers for anonymized review will tell you several things. First, the papers one chooses (and doesn't choose) to cite in a paper placed under review can very much be a way of "signalling" professionally. Generally speaking, reviewers will look for you to cite people *they* think are important--and, often enough, those people are Big Established names (since, all too often, people reviewing your paper are recognized names in the field themselves). So, what happens often enough in anonymized review is that authors are incentivized to cite and credit Big Names over Small Names or No-Names. I've heard some people say this pretty openly, in fact (that they choose to cite big names to "signal" to referees).

Consequently, I think the incentive structure of anonymized review generates the very problems this post alludes to (not to mention the fact that it incentivizes No-Names to hide their unpublished work but not so much Big Names).

On the other hand, It think the incentive structure with an "open" peer review system would be very different--tilted not in the direction of crediting Big Names but instead giving everyone a more equal opportunity to lobby for the *right* names to be credited, even before a paper is placed under review.

Why? Because in an open, public peer-review model literally anyone--Big Name, Middle, No-Name, or Outsider--can get in on the action, debating publicly (and anonymously, if need be) whether a new paper draft adequately credits the true originator of a given argument or idea. If anything would plausibly incentive proper credit-giving, I believe it is precisely this: leveling the field by giving the *profession at large* the opportunity to debate and correct for scholarly lapses before a paper is reviewed or published (rather than one or two reviewers).

I don't know about you, but if we moved to a more open peer-review model, I would very much be on the lookout for these sorts of things, advocating openly for corrections to improper scholarship--and I expect there are a good number of others who would be on the lookout as well. While some people might find that annoying--a kind of scholarly busybodiness--these are the kinds of things I think a healthy profession should have: a more public, transparent system of checks and balances to protect authors and ensure proper scholarship. And indeed, I would very much be thankful if such a system helped me improve *my* scholarship, helping me better cite and credit authors in the manner they are properly due.

[Douglas W. Portmore]

Hi Marcus,

As a graduate student coming out of a poorly-ranked PhD program, there was little incentive to read my unpublished work. But, due to our practice of peer-review, some really great philosophers (e.g., Derek Parfit and Stew Cohen) were "forced" to read my work either because they were an editor at a journal where I had submitted my work or because they had agreed to review it at the request of such an editor. The result was that I got a lot of valuable feedback on my work that I wouldn't have otherwise received. For these people wouldn't have read it but for this process being in place. And my graduate professors didn't provide the same quality or quantity of feedback that I received via the peer-review process. And no doubt part of it was that they had to interact with me face-to-face. As a result of this feedback, I was able to publish my work in some fairly prestigious journals and that led to people reading my work who wouldn't have otherwise bothered to do so. So, I wonder how would I end up receiving the same advantages on your proposed "open-online" peer-review model? Why would any great philosopher choose to read the paper of some graduate student at a poorly-ranked program? And why, even if they did, would such a person take the time to provide detail feedback on that work? If you've already addressed this issue in a previous post, you can feel free just to direct me there.

Also, in my experience as both an editor and a frequent reviewer, blind-review is indeed blind the vast majority of the time. There are times when I know who the author is and the editor wants me to review it anyway. And there are times that I have a hunch who the author is. But most of the time I don't know whether the person who wrote the article that I'm reviewing is a student or a full professor at an R1 university. And this ensures that keep on my toes and don't ever dismiss anything too quickly.

[Marcus Arvan]

Hi Doug: Thanks for weighing in! I think the anecdotes you offer are important in few respects.

First, they illustrate something many of us recognize--and that I certainly want to recognize: namely, that there are of course instances where our current peer-review system works well, benefiting our work and careers. Like you, I have had cases of wonderful reviewers and editors--people who took an unexpected interest in my work, read it charitably, helped me improve it immeasurably, and took a chance on publishing it!

However, I think the relevant question is not whether there are individual cases where everything works as it should, including cases where someone like Parfit takes an interest in your work as a result of our current review process (as an aside, I hear Parfit was a rare and exemplary person who routinely took interest and provided comments on work he was sent - which I think makes him admirable and heroic, but perhaps not representative enough to stake peer-review policies and practices on). The more relevant questions, I think, are which peer-review system--our current one or the more open one of math and physics--(1) operates better on average, and (2) which system makes the worse/more harmful errors when it does err.

In this post and many others (and in line with Heesen and Bright's empirical research), I've tried to argue that--anecdotes aside--there are systematic reasons to think that the open model is likely to work better on average and our current anonymized review system likely to make the worse and more harmful mistakes (as evidenced, I believe, in the current post).

Bearing that in mind, allow me to address the questions you raise: "So, I wonder how would I end up receiving the same advantages on your proposed "open-online" peer-review model? Why would any great philosopher choose to read the paper of some graduate student at a poorly-ranked program? And why, even if they did, would such a person take the time to provide detail feedback on that work?"

Here is my answer. I actually think it is more likely that a great philosopher like Parfit or Cohen would take an interest in your work in an "open" peer-review system than in our current one. This is for two reasons.

First, in our current system, whether your or my work ever ends up in front of someone like Parfit seems to me, in large part, a matter of luck. On the one hand, you first have to get lucky to submit your work to a journal with an editor like one of them (or, as in the case you describe, getting one of them assigned as a reviewer for a paper you submit). Then, in cases where they serve as an editor, whether they take a real interest in your paper may be a matter of how lucky you get with reviewers. If, for instance, you get unlucky with unsympathetic reviewers who pan the paper, then perhaps the editor involved may not give it the kind of close or sympathetic read they might have given it if different reviewers were involved.

In other words, while it is certainly *possible* for a submitted paper to end up in front of the right Big-Shot, and then get them interested in your work, it seems to me that a lot probably has to go right for that to happen.

As an analogy, I'd compare it to the way that talent scouts used to discover musical artists. In the old days, A&R people would troll popular music venues--but you had to get lucky: you had to be playing in the right club, on the right night, in front of the right A&R person, and have a great show. And if you were one of the many artists who got unlucky--who never lucked out in having the right talent scout in the audience who their music spoke to? Well, those are the artists you've never heard of.

The point here is that even if it's possible to "get lucky" in a particular system (as you may have with Parfit and Cohen), it's also possible to *not* get lucky. And so we have to once again ask which system is likely to perform better her on average.

So, then, how would you probably get noticed by people like Parfit and Cohen in the "open" system of math and physics? And why do I think there would plausibly be less luck involved?

The answer is that I have seen how things seem to work in math and physics. Things work like this: people post new work online. Work that is actually good then tends to get talked about openly: it is debated, shared by other people in the profession on social media, and so on. In other words, word of mouth builds about it--it spreads. At this point, if you are a Big Shot like Parfit or Cohen, you will probably be likely to hear about that work in one of two ways.

First, if you are on social media, you'll probably notice the chatter (this is what happens in math and physics)--and so you will then go check out the paper to see what all of the fuss is about. Second, if you are not online or on social media (as I hear some people in the profession are not), you will be likely to hear about it in the hallway, at conferences, or in a seminar. If, for instance, you are Parfit walking the halls of Oxford or Stew Cohen at Arizona, some grad student or other faculty member may be apt to say to say to you during a reading group, grad seminar, or personal discussion, "Have you read that new paper by Small Fry everyone is talking about on philpapers? You have to check it out. It's right up your alley."

All things considered, I think there is likely to be whole lot less luck involved here. To use a music analogy once again, talent scouts no longer just troll clubs looking for acts--as they recognize that there's too much luck involved in that (not to mention many scouts signing "can't miss" acts that totally miss!). No, what record labels do today (or so I hear) is just pay attention to which new artists get listeners and followers on Spotify, Tidal, etc. Why? Because they recognize that an open, crowdsourced method--paying attention to what people are actually listening to--is a better, more reliable way to find acts that are likely to sell lots of records.

My suggestion then is that the same likely applies in philosophy. You want to get someone like Parfit or Cohen to notice your work? It's *possible* in our current system of anonymized peer-review--but it's far more likely, I think (for reasons just given), in a open system where word gets around.

The same, I think, goes for feedback. As someone who follows physics, papers in physics get a great deal of online feedback--in part because that open system of peer review doesn't just involve one to three anonymous reviewers, but indeed anyone who happens to download and comment on the publicly available paper draft--including someone like Parfit or Cohen who would plausibly come across the paper by word of mouth.

[Peter]

Hi Marcus,

Thanks for the response. This seems plausible to me. I still have other worries about your proposed system (ones not directly related to this post, so I will leave them aside here), but I must admit, you are slowly winning me over.

[Douglas W. Portmore]

Hi Marcus,

Thanks for the detailed and thoughtful response. One remaining worry that I have is that the open system will not necessarily favor the best work but will almost certainly favor work that's shorter, less specialized, less technical, and otherwise more appealing to a wide audience. As your analogy suggests, "paying attention to which new artists get listeners and followers on Spotify, Tidal, etc....is a better, more reliable way to find acts that are likely to sell lots of records." But it is unlikely to be the best way to discover brilliant music that's not readily accessible to a broad audience. But, unlike record companies, the point of our peer-review system is not, I hope, to find the articles that will get the most "customers" but to find the articles that have the best insights.

[Marcus Arvan]

Hi Doug: Thanks!

Regarding your residual worry ("the open system will not necessarily favor the best work but will almost certainly favor work that's shorter, less specialized, less technical, and otherwise more appealing to a wide audience"), I think there are good reasons to think it is not likely to be realized.

First, it does not appear to be realized in math and physics (which use the open model). For example, perhaps the most famous papers in all of math right now are four-preprints by Shinichi Mochizuki that purport to solve the abc conjecture. https://en.wikipedia.org/wiki/Inter-universal_Teichm%C3%BCller_theory

These unpublished preprints total over 600 pages and purport to invent an entire new realm of mathematics that is so incredibly specialized, techincal, and novel that hardly anyone understands it. The papers have generated an incredible amount of interest and discussion among mathematicians for precisely these reasons--because they are experts *interested* in technical, specialized, sophisticated things, not merely math for mass consumption.

I think there is every reason to believe that things would work similarly in an "open" model of peer-review in philosophy--that is, that an open model will favor good work emerging and being discussed.

Sure, there would probably be some less technical papers that would gain a popular following (including interest among nonspecialists). But, first, I don't think that would be a bad thing, any more than it is a bad thing in physics (a discipline that has greatly benefited from popularization as well as interesting but ultimately flawed work by outsides, see e.g. https://en.wikipedia.org/wiki/An_Exceptionally_Simple_Theory_of_Everything ).

Second, and more to the point, I don't think there is any reason to think--given the interests of professional philosophers--that an open model would problematically favor shorter, less technical, philosophy "for mass consumption." For professional philosophy--in a quite obvious way--is not like Spotify. On spotify, the average listener is just your average person--someone with little expertise in music. In philosophy on the other hand (as in math and physics), the typical reader of philpapers pre-prints is going to be a specialist. And it is not as though philosophers interested in technical, specialized work--and able to discern good quality work from bad work--are going to suddenly disappear off the face of the earth or lose their interest in technically sophisticated work and ability to discern good work from bad work. No, in an open model philosophers would still be experts: there would still be the Derek Parfits, Stewart Cohens, Doug Portmores ;), and countless other serious professional philosophers who care about philosophical sophistication, quality, and so on--and would push online discussions of pre-prints in those directions, helping to recognize the best work out there.

Indeed, by my lights, the more open the system, the more likely this is to occur--as instead of leaving the judgment of a piece's quality up to a couple of anonymous reviewers (who can, as we see in the OP, get things very wrong), professional philosophers in the "open" system would be likely to do exactly what the mathematicians and physics do: debate the merits and deficiencies of new papers widely and openly--which I believe is likely to be at least as good a method for recognizing genuine philosophical merit (and probably better) than our current model.

[Douglas W. Portmore]

Thanks, Marcus. That's very helpful.

[Another]

Marcus,
I am inclined to agree with Douglas' first expressed view. I think our responses - and yours too - are largely a largely a function of our experiences. I feel I have had really fair treatment in the journal referee system. Of course there are particular papers with particular journals that may have gone not so well. But on the whole I feel despite the fact that for most of my career I was not at a high profile place, my papers were treated justly. Further, I have similar experiences to those Douglas reports. Senior important people in the field have said to me at conferences "I have read your paper ... it is great" ... or something of the sort. This has happened a number of times.

[Marcus Arvan]

Hi Another: Our responses may of course be influenced by our experiences. However, I do not think the arguments that I or others have given for changing peer-review ultimately boil down to personal experience.

As I have said, I have had some very good experiences in anonymized review, as well as some not-so-good ones. In this regard, from what I can tell I seem to quite typical. Like you and Doug, I have shared very positive experiences. However, I have also had some very negative experiences--and routinely encounter people reporting similar experiences on social media (such as capricious referees, absurdly long wait times, being deprived of credit for new ideas, etc.).

The arguments I have provided--and which Heesen and Bright provide--do not boil to mere anecdotes or experiences. As Heesen and Bright argue, there is literally *no* clear empirical evidence supporting the benefits of anonymized review, whereas there is quite a bit of evidence on the costs. Some of those costs are in turn exemplified clearly and disturbingly in the piece on economists I shared in this post. Finally, in the OP, other past posts, and my discussions in the comments section with Peter and Doug, I have tried to give detailed arguments for why various features and incentive-systems inherent in both models of peer-review favor the open model over our current one.

In short, while my background motivations for making these arguments may indeed have something to do with the experiences I have had personally and heard about second and third-hand--both good and bad, and in our discipline and other disciplines (math and physics)--I think the arguments themselves are *not* based on mere experience, but rather on a variety of converging lines of evidence: scientific evidence (of the sort Heesen and Bright point to), historical evidence (of the sort illustrated in the economics piece), and analysis of the features and incentive structures of rival models.

Like you and Doug, I have actually fared well in anonymized review. But that does not indicate to me that all is well, nor that we should content with the status quo given the overall balance of evidence. I advocate for change here because, when I survey all of the available evidence--and reason through it--the overall balance of evidence in favor of change seems to me clear (for many of the same reasons Heesen and Bright argue it is clear).

[And Another]

In Science as a Process, David Hull provides some interesting evidence about anonymous peer review in biology. Reviewing (many) referee reports for a journal on which he served on the editorial board, Hull found that it tended to be those who shared similar views as the authors of papers that were more critical in their referee reports. Hull then attributes this to the fact that scientists do not want papers in support of their view published that appear weak in argument. They thus are more critical of their allies.

[Marcus Arvan]

And Another: I haven't read Hull's piece, but judging from your description of it there seem to me several problems.

First, although it is an anecdote (and should be taken into account as such), it appears to a single editor engaging in post-hoc anecdotal without any control for potential bias (such as, potentially, wanting to find what he found). Unless I'm missing something (and please do let me know if I am, to be fair!), this is not an adequate design for detecting anything of statistical significance. Good empirical evidence normally requires multiple independent sources of evidence, control groups, etc.

Second, on that note, Heesen and Bright reference the empirical literature demonstrating little interrater consistency among reviewers, including one study of a top journal demonstrating that "“reviewers. . . agreed on the disposition of manuscripts at a rate barely exceeding what would be expected by chance.” (p. 18)

[Chris Stephens]

I've probably mentioned this before: as you probably know, philosophers of science have had access to the philosophy of science archive that Pittsburgh's HPS program hosts: http://philsci-archive.pitt.edu

But: it isn't set up for all the editorial and refereeing practices that you mention. However, it does have the advantage that if someone is worried about being scooped, they can post their paper there. It also makes the paper public in way that's a bit more than just putting it on your phil papers or home page. That would at least address some of the worries you raise.

I do worry that there are important disciplinary differences between philosophy and math/physics that might affect how well it would work for philosophers. It would be interesting to know if, say, history had set up and had a system similar to the one you propose, and to see how well it has worked for a more "humanities" discipline (Philosophy, of course, is really a hybrid discipline and only partially a humanities subject).

[And another]

Marcus
The study by Hull is more than anecdote, but I will leave it to you to find out.

[Marcus Arvan]

And another: Okay, will do!

[Marcus Arvan]

Peter: Thanks! :) I'd be curious to hear what your other worries are sometime.

[Peter]

Marcus,

I will leave my worries for another time (I want to go back and reread all of your other posts--my worries might be addressed already.)

For the moment, I have another question: let's suppose that you are correct. How do you think we might begin transitioning as a discipline to something like what you suggest? Do you think it might start best with some subdisciplines beginning something like this (I know philosophy of science already has something a bit like this)? Do you think this would work better than trying to get something together discipline-wide (perhaps through PhilArchive, for example)? It is an interesting problem since it involves both a change in behaviors and norms on the one hand and the creation of an infrastructure that is designed for this purpose. (It seems to me that PhilArchive is not quite suited to what you have in mind, at least not as it currently is, but maybe I am wrong.)

[Marcus Arvan]

Chris: Thanks for chiming in. Of course, we also have philpapers, which seems like it would be a natural fit for this function--though people don't use the way people use the pitt archive in philsci or the ArXiv in physics. Philpapers also doesn't currently have some important functionality that the ArXiv has, such as listing on a paper's page how many places it has been blogged about (along with links to discussions).

These are things I think we should think about (and philpapers should think about) moving forward. As I say in my reply to Peter below, I hope to write up a post on this stuff soon!

[Marcus Arvan]

Hi Peter: You sort of read my mind! I was just thinking this morning about writing up a post on this: on things we could do--as individuals and as a discipline--to transition in the direction of the "open online" peer-review model. I hope to add to this a few concrete things *I* am tentatively planning to do to promote such change.

As a brief preview, I think transitioning toward such a system will probably have to involve:

(1) People like me who favor and advocate for the system taking concrete steps that favor it, such as posting paper drafts online, blogging about unpublished drafts posted on the PhilArchive, and so on.

(2) People in positions of professional and institutional influence (e.g. the people who run Philpapers, etc.) taking steps to foster the new system, such as adding new functionality to philpapers and advocating for and acting according to new research norms (such as treating unpublished papers posted to the PhilArchive as the sort of thing one should be aware of, discuss, cite, etc.)

Anyway, this is just a preview of some possibilities. I hope to think things out a bit more as well as solicit ideas from the Cocoon's readers. But this is sort of what I am thinking: it's time to stop just talking about the "open" system but (at least for those of us who support it) taking active steps to realize it.

As a quick aside, I'm not inclined to think it would be best for specialty disciplines to do it. Philosophy of science has already done it (as you note), and it seems to have worked fine (in terms of people citing stuff posted to the philsci archive). But I think for it to *really* work it has to be more central to the discipline and centralized (as the ArXiv is in physics)--as I think it's only when we begin to take on new norms as *philosophers* (as opposed to just ethicists, etc.) that the new norms and practices are likely to stick. Which is another reason why I think philpapers/philarchive would be great. It's already probably the most widely trafficked site for philosophy research. By my lights, it should be our ArXiv, though much must be done to realize that promise!

[Peter]

Hi Marcus,

This might be a consideration for another post, but in your response to Chris you say "Philpapers also doesn't currently have some important functionality that the ArXiv has, such as listing on a paper's page how many places it has been blogged about (along with links to discussions)."

This makes me wonder whether math and/or physics has a different culture of blogging than philosophy does. Does a large percentage of academics in these fields blog about research? If so, is this a central reason for the success of the ArXiv system at this point in time? It seems possible to me to have a successful open system without a culture of blogging, but your comments make me wonder whether this particular open system is somehow intertwined with other aspects of academic culture.