Our books






Become a Fan

« Open Signups! Job-Market Mentoring Program | Main | What if your philosophical idea is the wrong length to work out properly? »

06/03/2017

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

reviewer

Marcus,
This may be somewhat unrelated, but I think academics have to realize that even if journals change, they are still not free from anonymous reviews.
I have reviewed - anonymously - post doctoral applications and grants applications, usually for European agencies (for the equivalent of US$ 100,000- US$2,500,000), as well as tenure files.
So anonymous peer review is not going away.
My own experience with journals is quite positive. I am not at a high prestige school. But I have published in some very selective journals.

Marcus Arvan

Reviewer: I absolutely agree! As I mention in the post, I've had some excellent experiences with peer-review too, receiving wonderfully helpful reviewer comments on some occasions. And of course grants, etc., should be anonymously reviewed. I just think our system could be much improved for everyone involved: for authors, editors, reviewers, and the profession more broadly.

Amanda

I think you point out a lot of interesting and accurate psychological problems with the peer review system.And I am in favor of changing the system in the ways you suggest for others reasons. Namely, because I think the alternative system would result in the best work being respected and discussed far more efficiently than the "anonymous" review system we have today. A problem with changing things, however, is I think many people have a love/hate reltionship with peer review and the love part creates a tendency to keep the status quo.

The hate part of the love/hate reltionship is obvious. Here is what I see as the love part. Even though those with connections have a huge advantage in the peer review system, about half the time or so anonymous review works and people have their papers published even if they are a "nobody" in the profession. Hence both those with and without connections get huge psychological boosts when their paper is published and that high is invigorating. Also, many people, I suspect, hate to admit that the game is rigged because that would make their passion (philosophy) and career somehow less valuable. Hence philosophers have a vested interest in the legitimacy of peer-review and I suspect there is a confirmation bias at work: I want peer review to reflect actual merit, because that would show that I work in a merit-based profession, and that is good. Hence I will see the evidence that suggests peer review is merit based and ignore evidence to the contrary

F.

Thanks for this, Marcus.

There may be a different kind of stress with the Phi-ArXiv system you discuss, especially if you are in a disadvantaged position in the profession or come from an unranked school and discuss non-mainstream ideas and have to post non-anonymously. This may even result in preventive self-censure.
On the contrary, people already famous or working at leading universities will capitalize even more on this.

I suspect that in math (I have no experience or training in physics) there are more objective standards in evaluating someone's work that would trump most other factors.

reviewer

I am curious to know how widespread the system you describe is used in math. I looked at the information page for the journal Non-linear Analysis. It is listed in Google metrics as the Top publication in the sub-category "Pure & Applied Mathematics".

follow link:
http://scholar.google.com/citations?view_op=top_venues&hl=en&vq=phy_pureappliedmathematics

The journal gave the following information, which makes clear that reviewers review anonymously:

"This journal operates a single blind review process. All contributions will be initially assessed by the editor for suitability for the journal. Papers deemed suitable are then typically sent to a minimum of two independent expert reviewers to assess the scientific quality of the paper. The Editor is responsible for the final decision regarding acceptance or rejection of articles. The Editor's decision is final."

see link:
https://www.elsevier.com/journals/nonlinear-analysis/0362-546x/guide-for-authors

Marcus Arvan

Hi F.: My understanding is that the ArXiv system does not have so much of those kinds of stresses, nor that it promotes self-censure.

There are a lot of papers defending non-mainstream ideas on the ArXiv. Further, work posted by "nobodies" is often discussed and published. To take one notable case, Garrett Lisi's ArXiv paper, "An Exceptionally Simple Theory of Everything", became a minor sensation for a time. Although Lisi's paper was proven mistaken, it received wide discussion by physicists, including 38 other ArXiv responses.

One of the beauties of the ArXiv system is that, if anything, it levels the playing field! If a big name posts a problematic paper (one that could otherwise pass "anonymized" review), people can and do criticize it on the ArXiv, drawing attention to its flaws. On the other hand, if a "no-name" posts a good or provocative paper, its merits may be discussed--not only by other big names, but also small names, etc. It's not a perfect system, but from what I can tell it is more advantageous to those disadvantaged in the discipline.

I have often heard it said that math and physics are more "objective" than philosophy. This is not at all clear to me, especially when it comes to theory papers in physics (as opposed to experimental ones). In any case, I still think the open/public review model would be better for philosophy. Just imagine if we had such a system right now. You--and anyone else--could go on philosophy's version of the ArXiv and criticize or defend anyone's paper you like. You would have to give arguments in doing so, demonstrating to other readers what is good or bad about the paper. They could then do the same.

Although no system is entirely free from bias (including anonymized review, where reviewers sometimes make bad arguments for rejecting papers or no argument at all!), my sense is that open discussion is the best way to determine whether a paper is a worthy addition to the literature.

Marcus Arvan

reviewer: I know the physics model better than the math one, but I've heard they are similar. My understanding is that although journals may still have anonymized review, papers are so widely distributed and discussed prior to submission that the anonymized review process is more pro forma than anything else (combining "anonymized" review at the very end of the process with open discussion prior to submission). That's more or less the system I think should exist.

Preston

I'm wondering how good ArXiv actually is at giving quality feedback to anyone but big names. My concern arises from something that tends to happen within citation practices in published philosophy.
Given the large amount of papers floating around, and finite time, people tend to read and cite the biggest names and those that have published in top journals. (Marcus, I know this bothers you, but its the way it is, and probably the way it will be until our teaching load gets reduced or some other radical change occurs.)

So if we go to an ArXiv system, why not expect people to just not look at (and thus not comment on) papers by people they haven't heard of a.k.a. junior people? It's not even necessarily irrational, even if it is unfair: If I know big name X's paper will (a) be read and discussed a lot, and (b) may be more likely to be of high quality (this will be a weak correlation, yes, but at least many people would endorse it), why wouldn't I be more likely to read the big name paper?

Anyway--maybe ArXiv has a way of avoiding this problem. But without anonymity, I don't see how it could.

Amanda

Well Preston I think the only way to know is to try. I suspect it would give lesser known people a better chance, and as you point out the big names already get all the credit. It would be nice to have some sort of trial system.

Skef

I broadly agree with the criticisms of the current system. But I am much less confident that an ArXiv-like system would work for philosophy. Examples of similar systems in use by a wider variety of fields would go a long way towards increasing that confidence.

Your argument seems to assume that systems of review are generic, and that the fields of Math and Physics have arrived at a better system. I find it more plausible that those fields have switched to an open review system because it is more suited to their publication standards and subject matter. It is hard to pin down in words what makes those two fields distinctive, but it clearly relates in part to much greater agreement on what constitutes a successful argument than exists in most fields. Math is of course strongly associated with the notion of proof, and while not every mathematics paper is a proof, the association does represent a more amorphous trend of agreement about what arguments do and don't meet mathematical standards.

Of course, you (Arvan) in particular may have reason to support such a switch because you also believe (and have argued elsewhere) that philosophical standards should be more like those of sciences. However: (1) there's still quite a gap between the standards of Physics and Biology, let alone Sociology, and (2) it's not likely that a shift to an ArXiv-like system would effect a shift to those standards. A cultural change like that would have to happen before the switch to a different system, or at least independently of it.

It's rather nebulous what makes a good philosophy paper. It seems fair to say that the two standards most widely agreed on are its contribution to further discussion (which can be loosely measured by citation rate) and "standing the test of time". Either of these are, of course, difficult to predict up front.

But more importantly, there is a general view that a central role of the current system is to identify what is worth reading. Journal editors complain most about having to sort through papers they do not feel are "ready". That problem does call for a different division of labor. But, as Preston points out, it is not at all clear why one would expect that a paper posted to a central repository would be read at all, unless it had some attention-getting property, such as being written by a known name, or coming out of a prominent institution.

Pendaran Roberts

I certainly can understand being frustrated with the current peer review system. It's incredibly slow, a high percentage of referees do a poor job, and too many journals don't provide any comments regarding or explanation of rejections.

I can't speak for how well community, non-anonymous review works in math or science, not having published in those areas, but I worry that for philosophy the system would be worse than the current system and for various reasons.

1. It would deter non-mainstream or alternative ideas. Imagine writing a paper on any number of contentious topics where you promote an unpopular view? What if it's an identity politics paper or something on ethics that's contentious like the value of mentally challenged people or any number of topics that are going to get people riled up. Are you going to want to have to go through a non-ananymous process? I don't think so!

2. It would favor famous or well connected people. Whose work is going to be read more: mine or famous philosopher x's? So, famous philosopher x is going to get dozens of comments, and I'll be lucky to get a few comments. I'll definitely get less. Famous or well connected people already have so many advantages. Do they need more?

3. Many people wouldn't benefit at all. For many early career philosophers, those not well connected or working on really hot topics, or... the only people who are going to read the work posted are going to be friends and colleagues. They could have gotten comments from them without this system. How do I know that for many early career folk that they aren't going to get tons of comments? Well just look at the citation rates in philosophy. We're a discipline that generally doesn't care that much about the work of others.

4. How do you ensure the quality of comments? Although the peer review system often fails in delivering authors' papers to expert reviewers, in theory at least editors are in a good place to determine who is qualified to review a paper. Let's say the community review system did work for some paper to provide the author with lots of comments. How is the author now supposed to weed through them all and determine which are from commenters that know what they're talking about and which are from idiots?

5. How do you stop people from stealing your ideas? Perhaps in math and physics it is rather obvious when an idea is stolen, but not so in philosophy. An idea can be something as simple as 'argument x requires presupposition y but...' Once someone sees that this is indeed the case based on reading your draft, they may be able to write a paper quite different from yours that nevertheless makes your paper mostly redundant.

These are just a few worries of many I have. Perhaps they can all be addressed. However, I just wanted to voice that I have concerns about the system Marcus proposes.

Pendaran Roberts

The peer review system in philosophy can be significantly improved with just one change!

The two biggest issues with our peer review system are

1. It's very slow.

2. Often the quality of reports is low.

Why? Simple. There is no benefit to reviewers for doing a quick and quality job, and no punishment for those who don't.

Imagine if Honda or Ford or... got the same money no matter how safe, fast, or good looking its cars were? We'd get cars just good enough to stop us from going mad and murdering all the automobile executives! hahaha!

The solution then is simple. Reviewers who do a quick and quality job should be rewarded for their work, and reviewers who do not should be punished.

There are many means by which reviewers can be rewarded and punished (sweets and electric shocks?), but I'll outline the most obvious and easiest to implement.

If a reviewer doesn't return a quality report within the allotted time, they aren't allowed to publish in that journal (or any journal?) for a set amount of time. However, if they do return a quality report within the set time, they get paid 50 dollars (calculated at 10 dollars an hour (minimum living wage) for 5 hours. A respectable bit of spending money.

Where to get the funds to pay referees? Easy. It costs 100 dollars to submit a paper (50 for each reviewer). This is a simple 'flat tax' system.

The only complaint I can foresee is that this system hurts unemployed philosophers, students and the part-time employed (thinking of you guys adjuncts!) who cannot afford to pay 100 dollar submission fees. Fair enough!

I thus propose a tiered fee system based on income (we already do this for taxes folks!). So, if you're one of the big shots in the profession who makes 200-400 thousand a year, your submission fees are very high (500 dollars lets say). If you're a grad student with a decent stipend, your fees are much much lower (40 dollars?). If you have no income, you can submit for very little (5 dollars?) or for nothing.

Of course the specifics of the tiers can be debated and implemented in a way that we can all agree on. I need not iron out all these details here nor could I or should I. My point is that our problems have easy solutions. It would just take some organization to force this system to be implemented.

reviewer

Pendaran,
How many people in our profession make $200,000-$400,000?
I think very few. I know some who do BECAUSE they are senior administrators, for example, presidents of private elite universities in the USA (Amy Gutmann). Surely not that many, though.
There are additional dangers of paying people to review papers. We are supposed to be a clan of scholars, not business people. This may be a false self-conception, but it might play an important functional role in curbing certain types of behaviours.

reality check

Many philosophy professors (employed as philosophy professors and not administrators) make over $200k. Amy Gutmann makes over $3.5 million.

Of course, the vast majority of TT philosophy professors in the US make $40-80k (not to mention contingent faculty). But make no mistake: our profession is a microcosm of income inequality.

Sources: http://www.thedp.com/article/2017/05/gutmann-salary-increase-2015 and various public university salary reports.

Pendaran Roberts

There are definitely philosophers who make over 200k a year. Yea there aren't many. Maybe a few dozen? I don't know. But that's not really all that relevant.

Clearly some in the profession make good salaries and others hardly get by.

So, we can design the system to be tier based to be more fair.

And the system as it is currently set up doesn't work well. I can't honestly think of any harm that could come from paying referees to do good work and having those who have good jobs front the costs for doing this.

Yea, I suppose it would be an additional expense for people, but you get what you pay for. If we're not willing to pay for peer review, it's going to continue to suck.

Amanda

I like that idea Pendaran. There is just no incentive to review, and so people do it slowly and often badly. I remember when every one made a huge deal about the $20 fee philosopher's imprint was charging. I think in the end they agreed to wave the fee for grad students, but then they seemed to have went back on that (quietly?) When I have submitted there they just asked for $20 straight up. I am fine with that, because as poor as I was as a grad student, I could afford $20, especially considering that I would not be submitting articles to phil imprint too often. In any case, when you consider that fees can be scaled according to income, I see no reason not to implement a paid system of reviewing which would impose real incentives on both ends. (Having to pay might also encourage less shoddy submissions).

Marcus Arvan

Pendaran (and Amanda): I don't think requiring people to pay for submissions is a good idea, for obvious reasons (it would disproportionately deter submissions by those in the most vulnerable situations in the profession). But I do think carrots and sticks for referees are a good idea. See http://philosopherscocoon.typepad.com/blog/2015/08/reforming-refereeing-carrots-and-sticks.html

In terms of Pendaran's and Skef's worries, I don't know how to evaluate them given that our discipline hasn't tried the public approach in math and physics. What I do know is that our current publishing model has all of the vices I've enumerated here and elsewhere, and that many of these vice appear to me mitigated in math and physics. I also tend to have more confidence in transparent procedures where everyone can take part, as opposed to non-transparent procedures dominated by a few select gatekeepers. Public debate enables concerns like Pendaran's and Skef's to be addressed *publicly*. For instance, if people tended to read and comment on works by Big Names, or react to politically unpopular arguments, others (like Pendaran and Skef) could step in and make their thoughts known as well. Once again, I think this would only improve the dissemination and discussion of research relative to our current model--especially if people increasingly abided by norms of citing and engaging with work posted on philosophy's version of the ArXiv (which, again, could begin and evolve in a grass-roots fashion my people--such as myself, Pendaran, Skef, and others--who have opinions on what is posted, what should be read, what should be engaged with, etc.

Skef

Perhaps it is more constructive to work backwards.

Bits are cheap, and although "bit longevity" isn't *as* cheap, it can be arranged. So suppose an archive for philosophy papers like ArXiv was established with a partnership with a library to make long-term availability more likely, and a standard citation system. Now any philosopher can "publish" any paper. (There would presumably be some level below which a paper can be declared spam, but that line could be well below student-paper standard.)

So, why isn't this system sufficient? What benefit do journals provide above this?

It seems that there are two main answers: 1) directing attention towards papers that are more worth reading, and 2) providing early evidence (prior to citations) of quality for career advancement.

Those people rating papers on the ArXiv site probably shouldn't be doing it *for* reason #2. I completely fail to see why they would just "naturally" do it for reason #1.

One thing that seems at best half-thought out about this proposal (and, indeed, the apparent current system in Math and Physics) is the archive being a step on the way to a journal slot. What is the point of a journal in this world? If some indicator is needed prior to citation rates, why can't it just be a marker (such as an "up-vote" or whatever) on the archive? Is it that we can't face a future without typesetting?

Marcus says "I don't know how to evaluate [my and Pendaran's worries] given that our discipline hasn't tried the public aprroach in math and physics". My suggestion was to look for a field with current evaluation standards closer to those of philosophy that is doing the same thing. It's of course possible that other fields just haven't taken the plunge for whatever reason, but I still find the explanation in terms of the more unified standards of those fields more plausible.

(Another factor here is paper length, the subject of a more recent post on this site. If relative dissertation length is an acceptable proxy for relative paper length, Math and Physics are distinctive by that metric as well: http://imgur.com/gallery/Gptq8Oh )

Amanda

Hi Marcus,

Given that most people do not submit that many papers a year, and given that it is completely plausible to charge as little as $5 per submission, I think it is far from "obvious" that charging for submissions would discourage the most vulnerable from submitting work.

Amanda

PS: I meant $5 would be the charge for grad students, adjuncts, etc. -with full professors paying more.

Marcus Arvan

Amanda: Fair enough. I just worry about requiring people in an already-underpaid profession to pay for even more things than they already do.

Grad students and adjuncts already have to pay for job-applications (viz. interfolio, etc.), conference travel etc. Further, I know some tenure-track professors who make something like $40K before taxes, who also have to fund a lot of their own conference travel.

In my view, we should not be looking for yet another way to nickel and dime philosophers--especially not when it comes to journal submission, where for-profit journals already rely on free labor (of authors, reviewers, etc.) to make a profit.

Pendaran Roberts

Marcus, you get what you pay for. When so many referees are under employed or unemployed, not paying them is a recipe for the kind of poor quality reports and slow turn around we all know so well. They just have no incentive! I mean a lot of people just never referee, not seeing the advantage to doing so.

I think we could design the system so that tenured faculty payed for most of the cost. To whatever degree those less fortunate have to pay refereeing fees, keep in mind that they also have the opportunity to collect refereeing fees.

So, my system would give the disadvantaged an opportunity to make money. I've refereed 10 times. So, under my system I would have earned 500 dollars. My referee fees would be very low given my situation, probably 5-10 dollars. So, I'd either be at an advantage financially or about even.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)

University of St. Andrews Grants


Job-market reporting thread

Current Job-Market Discussion Thread

Philosophers in Industry Directory

Cocoon Job-Market Mentoring Program

Categories