A propos our recent discussion of whether philosophy journals should have and enforce clear peer-reviewing standards, I came across this 2011 paper by Carole J. Lee and Christian D. Schunn comparing philosophy's peer-review practices to psychology's (thanks to Feminist Philosophers for drawing my attention to it). The paper contains the results of:
- A study coding 423 reviews for 221 papers submitted to the 2007 Cognitive Science Society conference)--112 philosophy reviews of 53 philosophy papers and 311 psychology reviews of 168 psychology papers--for negative and positive comments, as well as for "inflammatory comments" (p. 358).
- The results of surveys of psychology and philosophy journal editors (pp. 360-)
Although I am not in a position to say much about the studies' methodologies (though I would certainly invite those who can to comment!), here are some of Lee and Schunn's rather striking findings:
- Nearly 30% of philosophy reviews contained at least one inflammatory comment, compared to only about 15% of psychology reviews (p. 359).
- The total percentage of philosophy papers receiving inflammatory comments (13%) was also about twice that of psychology papers (6%) (ibid.)
- About three times as many philosophy papers contained multiple inflammatory comments (4.5%) as psychology papers (1.6%). (ibid.)
- 41% of philosophy papers were rejected, compared to only 20% of psychology submissions. (p. 360)
- "[In philosophy journals] A single negative review has more power than a single positive review on editorial determinations. About 40% of editors ‘‘never’’ or ‘‘rarely’’ accept a paper receiving a single negative review (of these editors, 80.0% report sometimes relying on a single review)." (ibid.)
- Philosophy journal rejection rates averaged 92%, compared to a mean of 78% for psychology journals and 20-40% for physical science journals. (pp. 360-1)
- "About 63% of general philosophy editors reported sometimes making determinations on the basis of a single review." (p. 357)
- "Approximately 25% of [philosophy] editors reported sometimes relying on three or more reviewers (though at least half of these editors remarked that this was not normally the case)"(ibid.)
- "In contrast, journals published by the American Psychological Association moved from a two-reviewer system in the 1950s to a three- to five-reviewer system in the 1990s." (ibid.)
Given that my wife works in psychology, I can personally attest to some of these differences. Thoughts?