Which criteria do you use as a journal referee?

In our newest "how can we help you?" thread, a reader writes:

I am a junior faculty member and started to receive referee requests. I am curious about people's criteria for recommendations (acceptance, minor revision, major revision, and rejection, among others).

In addition to any general thoughts, I am particularly interested in hearing your thoughts on the following two questions. First, do you take the "quality" of a journal into consideration? For example, is it reasonable to think, "this paper is not good enough for Phil Review or PPR, but for this journal, sure, it should be accepted"? I feel that this sounds pretty normal, but I also know that people do not like journal rankings. If the quality of a journal should be considered, what is your standard? Do you implicitly or explicitly follow a specific ranking such as the survey by Leiter?

Second, is there a principle that you follow when you make your recommendations? To be honest, the rejections that I received usually raised some objections to my arguments. Is it a good principle that if you can find an objection to the author's main argument, then you should (probably) reject the paper? Do you recommend direct acceptance only when you cannot find a good objection? Thank you!

Good questions! Another reader submitted the following reply:

[E]veryone follows different criteria for their verdict. But I often take the quality of the journal into account while deciding whether my comments and feedback mean accept/r&r/reject. This is not because of the quality of the paper, but because of the ratio of space available in those journals to the volume of submissions. This naturally implies a higher level of selectivity than perhaps otherwise merited. Re. how I evaluate my accept/r&r/reject verdicts: I do not take any objection to an argument to be a reason to reject a paper. A straightforward rejection usually takes me to believe that no small amount of work could salvage a novel, interesting and close-to-being-sound argument/idea from the paper. Sometimes this is an objection to the main claim of the paper and sometimes it is about how well-formulated the claim and the case for the claim in the paper are. If the paper is well-structured and interesting, but you have a prominent objection to one of the main parts of the paper, I think it usually worth giving the author to revise the paper and after the revision, if the problem is still there, then reject it (the caveats about journal rankings apply here as well).

And one other reader wrote in:

(1) I usually only really consider originality and specialisation in this. So, if I review something for the British Journal for Philosophy of Science, I recognise they are looking for groundbreaking work and am hesitant to recommend a decent and correct but not very novel paper. Or, if I review for Foundations of Physics I realise that they are more open to somewhat technical stuff than, say, Synthese. But I wouldn't otherwise make a difference in 'quality'. (2) I definitely don't think only papers without any potential objections to them deserve acceptance. The distinction I try to make is: is this an objection that should really be addressed in the paper, or is it an objection that could itself form a significant part of the literature? If it's the former, then I would urge a revision, or in some cases a rejection. If it's the latter, I am happy to accept the paper - the fact that someone else can object to it in a way that is itself of interest only proves that the paper under review pushes the literature forward in some way!

What about the rest of you? Which criteria do you use as a journal referee? And do you tailor your standards to different journal types or rankings?

 

Comments

[Daniel Weltman]

I don't think much about the journal when refereeing. I try to accurately describe my thoughts on the paper so the editors can make a decision. I assume the editors of more selective journals will be more selective than the editors of less selective journals, but this doesn't change what I write, just what I expect to happen based on what I write.

For some discussion on this topic see: https://philosopherscocoon.typepad.com/blog/2021/04/tailoring-referee-standards-to-journals.html

I do not follow a single principle when making recommendations, except negative principles, like "the existence of one or more objections does not mean I will recommend rejection" or things like that. In general I am looking to see if the paper adds to the conversation, if it addresses the relevant literature, if it accurately describes that literature, if the paper's arguments make sense, if the paper's scope is appropriate to its length, etc. There's a lot to say on the topic but generally I think the best advice is "write what you would want if you were an editor and also keep in mind what you would want as an author."

[Referee 1]

The OP notes that the papers they have had rejected have included, in the referee report, some objection. That is as it should be. The instructions from journals explicitly say that referees are supposed to give reasons for their assessments (good or bad). So if I am recommending a paper be rejected, then I need to identify faults with the paper that warrant my asessment. Likewise, when I recommend publication I try to explain why I think the paper is a novel contribution to the literature. But, with that said, as one of the others imply, I have recommended acceptance for papers where I disagree with the conclusion. And some of these are in print, in the journals for which I refereed (for example, Erkenntnis). So papers are not being rejected because referees disagree with your conclusion (at least not papers I referee). they are being rejected because of serious flaws in the argument, or presentaion, or for lack of novelty (they do not push the debate forward in any way).

[Ian]

I'm early in my career and have only done a modest amount of refereeing, but I have refereed for both high prestige generalist journals and more specialty journals. My sense of the reputation of the journal doesn't affect my recommendations. For generalist journals, I'd be more inclined to say something to the editors about my sense of whether the argument is likely to be of interest to a more general readership or whether it is really only likely to be of interest to specialists (which usually comes down to how the argument is pitched), but I would never think to myself or make recommendations on the basis of thoughts such as, "This argument is strong enough for a C journal but not for an A journal."

Ultimately it is the responsibility of journal editors to make decisions about whether a paper is appropriate for their journal. As a referee, my role is to give my sense of the quality of the paper and my sense of the significance of the contribution that it makes. My role isn't to determine if that contribution is significant enough for the journal in question.

I also don't make my recommendations merely on the basis of whether or not I have objections. I can think a paper is a tightly-argued, well-written, novel contribution to the literature and think it is totally wrong. I recommend rejection when I think that a paper doesn't make much of an original contribution, is poorly written or poorly argued, or is subject to what I think of as a devastating objection. An objection is devastating and becomes grounds for a recommendation of rejection when, e.g., the argument relies on a misrepresentation of the problems or views that it deals with in a way that I doubt revisions could help, it has a demonstrably false premise (i.e., a premise that isn't merely philosophically or interpretively controversial but clearly and demonstrably false), etc. In short, if I think that a paper is tightly-argued, well-written, and a novel contribution to the literature and my disagreements with it fall within the realm of my sense of reasonable philosophical or interpretive disagreement, then I'll recommend Accept or R&R, depending on how much I think a little more work could improve the paper. For disagreements that fall within the realm of reasonable philosophical or interpretive disagreements, I would rather see them hashed out in the literature than kept from the larger philosophical community by gatekeepers who, for reasons that I don't understand, seem to want to see only arguments with which they agree in the literature.

[lnwh grad]

I cannot forget about a phrase a referee kindly provided in a revised version of one of my papers: 'Any further disagreement should be settled in print'. I feel like in philosophy this wisdom too often goes ignored or not heeded.

[Michel]

I'm happily surprised to discover that I'm not alone in ignoring journal "quality" when formulating my verdicts. I think it's especially important for my subfield, because it's one that's seldom published in the higher-ranked generalist journals (with some notable exceptions!). Being particularly insistent on this would just reinforce the situation. But also, if I'm honest, I don't have a good, fine-grained sense of the quality distinctions between most generalist journals anyway (and I'm skeptical there is one). I've internalized the ranking, but it's entirely based on what others have said on the surveys rather than my own judgements of quality, since for most journals I either read almost nothing printed in their pages, or only those articles which are in my specialty (and there, I see no appreciable difference between its being published in, say, PR or the top specialist journal).

Plus, if I was being consistent, then it seems I should be rejecting some papers for being too good for their target venue, which just seems silly.

No, instead, I let the author and the editor decide whether the paper is a good fit for the venue (with a partial caveat, like Ian's above: if it's a generalist journal that seldom publishes in my subfield, I may recommend the author do more to explain the background of the issue, whereas for specialist venues it's fine for the author to just dive right in). What I decide, essentially, is just whether the paper is of publishable quality, and how much work is realistically required to get it to that point.

As a result, I'm generally a pretty permissive referee. My average verdict is an R&R (if rejection = 0, R&R = 0.5, and acceptance = 1, then my average decision is 0.67). I don't think one or more objections are sufficient for rejection; I'm happy to see work published that I disagree with entirely. If that were the standard, I would never recommend acceptance.

Generally, I recommend rejections when the argument doesn't work and the work required to fix it would be too extensive to reasonably perform in a few months/would result in a very different paper, when the scholarly engagement is seriously lacking (e.g. the paper cites a few big papers from the '70s and nothing more recent, or entirely ignores the fact that the debate is one that is not unique to them), when it's crackpot nonsense, when the scholarly contribution seems too negligible, etc. It's then up to the editor to decide whether they want to invest in that kind of process to publish this kind of work. And that seems like something they should know a lot better than I can.

I try to referee others' work the way I want to see mine refereed. I try to recognize that most of the objections I may have are not devastating; they're just sources of disagreement. And sources of disagreement are good fodder for another paper. As an author, when I receive a rejection that appears to be based on relatively minor, easily addressed objections, it just feels petty and frustrating. I try to spare others that sort of frustration.

[TheoristOfValue]

All good advice here.

I don't take journal prestige into account when rendering a verdict. In part this is because the quality of the work appearing in "top" journals (e.g. Ethics, PPA, Nous, PPR) has declined in recent years. They're no longer publishing the best philosophy (if they ever were). So even if there were some theoretical reason to try to "match" one's verdict to the quality of the journal, this is no longer a concern.