Reader query: best practices for reviewers?

A reader wrote in the other day asking if know of any online discussions of "best practices for journal reviewers." Now, we have discussed some issues regarding reviewing before. There is also Thom Brook's paper on SSRN, "Guidelines on How to Referee." Finally, there are some online resources for best practices, including the APA's own statement. However, given that views on best practices can evolve over time, different practices may be "best" for different academic fields, and different people can have different views on what best practices are, I thought it might be a good idea to start a discussion here. What are "best practices" for journal reviewers in philosophy?

I think one good way to answer this question is to break it down into more specific questions:

1. What sorts of things do reviewers you have encountered do that you consider to be good practices?
2. What sorts of things do reviewers you have encountered do that you consider to be bad practices?
3. Given answers to (1) and (2), what exactly should be expected from reviews?
4. How should journal editors enforce those expectations?

Comments

[Ken]

Here is a bad practice: If you reviewed a paper for one journal and then receive the paper again to review from another journal, decline it. In small fields this happens, a lot, and philosophers often just recycle reviews for some unknown reason, and editors often just accept them. The reasons for this strike me as too obvious to mention.

[Elisa Freschi]

1 and 3: good practices and what to expect: Do a *honest* review, one through which the reader will understand a. what the book is about, b. whether the book makes an important contribution and is worth reading.
2: bad practices: My impression is that most US reviews are too positive, so that all books sound as if "the field will never be the same after them" (thus failing to convey b). By contrast, many European and Japanese reviews are too critical as if spotting mistakes were the only way to show that one has really read the book. I am very much in favour of finding real flaws, but cannot see any advantage in lists of typos and the like.
4: Does not seem to be so difficult to check. Whenever I write to prospective reviewers, I send in advance a short text explaining them what is expected from them.

[Perplexed]

Elisa,

I am a bit perplexed by your comments. I thought Marcus was asking about refereeing, not book reviewing. Is that right, Marcus?

[Marcus Arvan]

Hi Perplexed: Yes, my post asks specifically about journal refereeing. Be that as it may, I appreciate Elisa's chiming in. It is an interesting side-issue to compare practices in journal reviewing to book review practices.

[Jerry Green]

Just did a ref report last night, so glad to have to think about this explicitly. Some good practices, which I'll put a bit more unapologetically and unnuanced than I probably should:

1) Finish your report in no more than a month. Let's be honest: it takes forever to do a report because we put off doing it, not because it takes a long time to do. I keep a timesheet, and mine take around 4-5 hours. If you don't think you can find a free afternoon to do a report in the next 4 weeks (6 tops), decline the request. But since refs are always in short supply, better to find the time instead.

2) Write a full report. I use a template with different sections, covering contribution, scholarship, method, and content. Say something on each, even if its short.

3) Any judgment should be followed by a because clause. So, don't just say "I find X unpersuasive' or "I don't think this argument works". If you can't clearly articulate what the problem is, odds are the problem isn't as bad as you think or you're reacting to something else.

4) Pay attention to bibliography. A journal paper is a work of scholarship, not just an interesting argument. A paper that doesn't engage with the relevant lit is no better than R&R, a paper that doesn't even cite relevant lit is a rejection, even if the argument is good.

5) Don't assume that 'unpersuasive' -> 'rejection', especially in the areas in which you have strong views. A paper can be worth publishing without winning you over, for instance if it stems from a plausible starting premise that you disagree with but can see the value in.

6) Be constructive. You don't have to explicitly tell the author exactly how to fix the problems with their paper, of course, but concrete advice is important. For instance, if they missed a relevant article, say what it is rather than just 'Doesn't engage sufficiently with the literature'. If there's an objection worth considering, present it for their consideration rather than just saying 'Argument has obvious objections'. Scholarship is an enterprise of group progress, and referees should play their part.

There's probably more to say, but you get the idea. General principle, if there is one: Do the kind of ref report you would like to receive yourself.

How to enforce this? No idea. Probably naive to think that we should be doing the right things as individuals without needed to have the rules enforced by someone else.

*disembarks from soapbox*