Philosophy's peer-review practices: some comparative data

A propos our recent discussion of whether philosophy journals should have and enforce clear peer-reviewing standards, I came across this 2011 paper by Carole J. Lee and Christian D. Schunn comparing philosophy's peer-review practices to psychology's (thanks to Feminist Philosophers for drawing my attention to it). The paper contains the results of:

1. A study coding 423 reviews for 221 papers submitted to the 2007 Cognitive Science Society conference)--112 philosophy reviews of 53 philosophy papers and 311 psychology reviews of 168 psychology papers--for negative and positive comments, as well as for "inflammatory comments" (p. 358).
2. The results of surveys of psychology and philosophy journal editors (pp. 360-)

Although I am not in a position to say much about the studies' methodologies (though I would certainly invite those who can to comment!), here are some of Lee and Schunn's rather striking findings:

• Nearly 30% of philosophy reviews contained at least one inflammatory comment, compared to only about 15% of psychology reviews (p. 359).
• The total percentage of philosophy papers receiving inflammatory comments (13%) was also about twice that of psychology papers (6%) (ibid.)
• About three times as many philosophy papers contained multiple inflammatory comments (4.5%) as psychology papers (1.6%). (ibid.)
• 41% of philosophy papers were rejected, compared to only 20% of psychology submissions. (p. 360)
• "[In philosophy journals] A single negative review has more power than a single positive review on editorial determinations. About 40% of editors ‘‘never’’ or ‘‘rarely’’ accept a paper receiving a single negative review (of these editors, 80.0% report sometimes relying on a single review)." (ibid.)
• Philosophy journal rejection rates averaged 92%, compared to a mean of 78% for psychology journals and 20-40% for physical science journals. (pp. 360-1)
• "About 63% of general philosophy editors reported sometimes making determinations on the basis of a single review." (p. 357)
• "Approximately 25% of [philosophy] editors reported sometimes relying on three or more reviewers (though at least half of these editors remarked that this was not normally the case)"(ibid.)
• "In contrast, journals published by the American Psychological Association moved from a two-reviewer system in the 1950s to a three- to five-reviewer system in the 1990s." (ibid.)

Given that my wife works in psychology, I can personally attest to some of these differences. Thoughts?

Comments

[Wesley Buckwalter]

It is utterly clear to me that philosophy desperately needs to have and enforce clearer and more professional peer-reviewing standards. We can do much better to improve this as a discipline.