In the comments section of my recent post, "Reviewers as ghost authors - a strange worry?", Wesley Buckwalter raised some interesting questions about the nature of peer-review and the main job of referees in that process. I would like to continue that conversation here, as I think our back-and-forth raised some interesting issues I think may be worth further discussing.
Wesley's view, in brief, is that, "the main job of referees is to ensure the minimum standard necessary for publishing", not help authors improve their papers. Wesley added, "I flatly deny the purpose of peer review is to help authors with papers. The purpose of it is to evaluate papers. Issuing an evaluation might and maybe should end up helping authors when done well, but in my view, as a consequence of its main function."
As I explain below, I think this is a natural and plausible way to understand the nature of peer-review and referee's "main job." However, I also think there are normative/moral grounds to favor an alternative model--one in which "the main job" of referees is twofold:
- To justify an editorial recommendation (e.g. reject, revise-and-resubmit, accept).
- To provide critical feedback in the recommendation that the author may use to improve their work.
Before I provide the grounds that I think favor this model, I think it is worth noting that there is at least one influential journal in another field that explicitly endorses it. Consider the following editorial policy of Cognition, a leading journal in psychology (my emphases):
We strive to publish the very best research in the cognitive sciences, with a particular focus on those papers that introduce novel experimental and theoretical advances, and that will be of interest and relevance to our wide readership. However, we cannot do this without the help of the many reviewers that Cognition contacts each year. Reviewing is a somewhat thankless task that takes considerable time, and no doubt we would all much rather spend that time writing our own papers rather than reviewing the papers of others. Probably we would all much rather say ‘no’ to each and every reviewer request we receive. And yet the expertise of the reviewers, and the quality and timeliness of their reviews, helps define a journal.
Reviewers have a responsibility both to the science and to the authors who are trying to advance that science. This responsibility includes helping the author better his/her paper and, if necessary, better his/her science...
Too often, we hear our colleagues (or even ourselves) refer to their experiences of the editorial process (across a range of journals) as ranging from unconstructive to confrontational. If the author can respect the editorial process as being cooperative and constructive, rather than confrontational, the journal as a whole, and its reviewers, will be held in greater respect.
Your role in the editorial process is to determine, as an expert in the field, whether the paper advances the field sufficiently to merit publication in Cognition, and whether there might be ways in which the paper (and the impact it might have) could be improved.
Cognition's editors suggest here three normative reasons to move to a more "helpful" model of peer review:
- It will plausibly lead to better science, by helping people improve their work.
- It will plausibly increase the reputation of the journal, in part by leading better work to appear within its pages.
- It will plausibly lead researchers to have a better professional experience, by fostering a professional culture that is cooperative and constructive rather than confrontational.
I think these points are worth taking seriously, and that there is a fourth reason to favor a more helpful model of peer-review, as well. Before I turn to the fourth reason, let me briefly comment on these three. My own experience is that helpful reviewing can indeed improve one's work immeasurably. The two published papers of mine that I am proudest of both became what they are because of really detailed, constructive reviewer comments. The papers in question probably would not exist today--they would probably not be published at all, or, if they were, they would almost certainly be inferior pieces compared to what they are--if the reviewers in question had not taken the time to provide the detailed, helpful comments they did. And I suspect I am not alone! Helpful reviewer comments are just that: helpful means to improve the quality of one's work--and, by extension, the work that appears in the literature.
This also speaks to point (2). By helping authors improve their work (by, say, not merely accepting a paper that meets a "minimum standard of publication"--but instead issuing an R&R with comments on how it could be a better paper, one that would be far better than the minimal standard for publication)--helpful reviewers improve plausibly improve the quality of work that appears within a given journal.
Finally, I think point (3) is pretty plain. As I explained in my recent post detailing my time at the Pacific APA, I kept happening upon conversations in which people expressed dismay at how "unhelpful" (some) reviewers (and editors) are. These professional frustrations, in my view, are not a good thing. And they stand in stark opposition to my experience with journals in another field: psychology (the same field as Cognition). My experience with psychology journals has been wholly positive. Even when my papers have been rejected, I have without fail received prompt, clear, helpful comments from reviewers (and, in cases of desk-rejections, from editors). One can only imagine just how much more pleasant--and yes, productive--our discipline might be if we had similar norms of "helpfulness."
Now, there are two related concerns one might have about such norms. First, providing detailed, helpful comments is time-consuming for referees. Second, or so one maintain, authors have plenty of opportunities to get feedback elsewhere--from colleagues, etc. However, I find neither of these concerns persuasive. Let me explain why, working backwards from the second concern.
Do authors really have plenty of opportunities to get feedback elsewhere? The short answer is this: it depends on one's professional situation. Consider, for instance, the following comment by Dan Kaufman at this recent discussion at Leiter Reports of how much time reviewers should spend on reviewing papers:
I was a Managing Editor for years, and I completely disagree with ['Out to pasture's comment]. For one thing, there is the assumption that everyone has access to the kind of networking described there, which is simply false. Many, many of us are working out in the sticks, in tiny departments, community colleges, departments merged with other subject areas, sometimes in quite physically isolated areas. (If you think it's as simple as just emailing Jerry Fodor or Stephen Schiffer and asking them to please give you their opinion on the paper you are working on, in Phil Mind or Phil Language, it isn't).
As a philosopher working in a very small department at a teaching oriented school, I cannot help but identify with Kaufman's points. When it comes to the ability to receive feedback, researchers in small departments like mine are at a vast disadvantage as compared to individuals working in large, research oriented departments. I know this from experience. Prior to coming to Tampa, I had firsthand experience in a large department (UBC) similar to Wesley's (Waterloo). While I was at UBC, the opportunities I had to get feedback were tremendous. All I had to do is walk down the hall to ask so-and-so whether they'd be willing to read one of my papers, and they usually would. Not only that: our department at UBC had people who specialized in basically every major area of philosophy--so, if I had a question or needed feedback, chances are there was someone with real expertise to I could get help from. This sort of opportunity simply isn't available in small departments. Indeed, many times over the years, I have written people I knew by email to solicit/trade feedback--yet, with a few exceptions, it rarely worked very well. Which this tends to leave someone at a small school like mine with roughly two sources for feedback: conferences and journal referees. Yet, I can get funding for only one, or maybe two, conferences per year--and, as we all know, conference feedback is not always that useful either. What has been incredibly useful for me--indeed, indispensable over the years--is helpful feedback from referees. Indeed, feedback from referees has, by and large, been the only means for feedback that I have been able to consistently draw upon--and, as I mentioned above, I think it has played a critical role in helping me not only publish, but publish better work than I otherwise would have. And, following Kaufman's remarks, I suspect I am far from alone: that there are many people like me out there who basically have few places to get feedback from except from journal referees.
So, here's the thing. Yes, more "helpful" norms for reviewing may be more time-consuming for reviewers. But, in my view, it promises to:
- Produce better philosophy for all of us to enjoy.
- Improve the prestige of journals that use the norm (e.g. lower-ranked journals benefit from better work appearing in their pages)
- Improve the overall professional experience of peer review (making it seem less arbitrary, less oppositional, more constructive, etc.)
- Give those who may have little access to peer-feedback (those in adjunct positions, small schools, etc.) adequate opportunities to receive feedback, lessening serious inequalities in opportunity in the discipline (i.e. those at big research schools can easily get ample feedback, those at small out-of-the-way schools cannot).
Finally, although it might cost referees time and energy in one respect (requiring them to spend more time writing helpful comments than they otherwise), it would plausibly save referees time in a more indirect way: by ensuring that more papers in the "peer-review pipeline" are better papers. Indeed, this seems to me an underappreciated problem with currently prevailing practices in philosophy (e.g. rejections without comments, or with unhelpful comments): this status quo simply encourages authors to send bad/problematic work out again, with few (or no) improvements--once again requiring new reviewers to spend time reviewing the same paper an earlier journal had rejected.
So, I think, we have ample reasons to prefer the "helpful" model of peer-review to the "referee's main job is enforce minimum standards of publication" model that currently seems to prevail. At least that's how it seems to me at the moment. Maybe I'm wrong. It wouldn't be the first time! ;) Anyway, what do you all think? (Final note: thanks again to Wesley to beginning this conversation. His comments really got me to reflect on why I'm inclined to favor the "helping" model. Thanks, Wesley!).