Tailoring referee standards to journals?

In our newest "how can we help you?" thread, a reader asks:

I have a question about writing referee reports and matching one's standards to the selectiveness of the journal.

I'm relatively new to writing referee reports and am finding it difficult to know whether and how to match my standards of evaluation to the selectiveness of the journal. To what extent do others try to do this, and could anyone recommend any specific strategies? I feel like I just find myself asking whether the paper adds something important to the scholarly discussion as a whole (and whether it does so in a clear, effective way, etc.), and I don't really have any specific strategies for assessing things differently based on the particular journal's standards.

Great question, and I'm curious what readers think. I've refereed for journals all over the map (top 5 journals, unranked journals, etc.), and I definitely use different standards. I always try to hold authors to standards of soundness and cogency, for obvious reasons. So, for me, the main issue in terms of tailoring my standards to journals is my sense of the piece's overall significance. For example, I tend to be a bit puzzled when I see a paper in a top-5 journal that makes what I take to be a very small move in a well-worn philosophical debate. This isn't to say that small moves aren't worth publishing (surely, they can be). It just strikes me that particularly selective journals should be in the business of publishing particularly noteworthy works that make major contributions to ongoing debates, and that smaller, less significant moves probably belong in other journals. Is it possible for a very small move to make a major contribution? Sure, Gettier's famous paper in Analysis is a perfect example. But papers like this strike me as far and away the exception, not the rule. 

Anyway, that's my main deciding factor for tailoring my refereeing standards to journals. But, having read such journals, my sense is that other referees may tailor their standards in different ways. For example, some particular journals seem to place a priority on formalizations (viz. formal logic and decision theory). Personally, I think formalizations tend to be overused in philosophy--but I recognize this view is idiosyncratic to me. Anyway, what do you all think, particularly (but not only) those of you who have reviewed for different kinds of journals? How do you tailor your standards to different journals? Or, alternatively, how do you think referees should tailor their standards?

Comments

[Douglas W. Portmore]

I just want to second what Marcus said. This seems exactly right: "I always try to hold authors to standards of soundness and cogency, for obvious reasons. So, for me, the main issue in terms of tailoring my standards to journals is my sense of the piece's overall significance. For example, I tend to be a bit puzzled when I see a paper in a top-5 journal that makes what I take to be a very small move in a well-worn philosophical debate. This isn't to say that small moves aren't worth publishing (surely, they can be). It just strikes me that particularly selective journals should be in the business of publishing particularly noteworthy works that make major contributions to ongoing debates, and that smaller, less significant moves probably belong in other journals."

[postduck]

"It just strikes me that particularly selective journals should be in the business of publishing particularly noteworthy works that make major contributions to ongoing debates, and that smaller, less significant moves probably belong in other journals."

I want to push back on this a bit. It sounds like a decision for editors at the desk-reject stage, not for reviewers. The way I see it, editors decide what topics or debates are suitable for their journal. If they think one topic should get more attention, or that another topic has become stale, or that another is too niche, they can just elect not to send it to reviewers. That's part of their power in the profession: they can influence what sort of work gets considered "top-tier." Reviewers, on the other hand, should tell editors whether a paper on a given topic adequately engages with the relevant literature, makes a novel contribution, etc. If an editor sends me a paper on a niche topic that's within my area of expertise, I tend to assume it's because the editor thinks that niche topic is suitable for the journal - otherwise, why send it to review?

(One bit of evidence that this is how things currently work is that top journals in philosophy seem to have really high desk-rejection rates already.)

There are also top journals that have a post-review decision-making stage involving the whole editorial board or the EiC instead of just the EA. That's another chance for them to make a decision about journal fit. There's no need for reviewers to throw up another roadblock.

On a more normative level, I think maybe it's not so good to just internalize and reinforce the current journal hierarchy, which is a bit of an old boys' club. I personally find that many of the most exciting philosophy papers that I read are not in top-5 journals, and that those journals have a fairly conservative, traditional, and narrow editorial focus. I'm certainly not going to recommend "reject" for a paper that I'd otherwise recommend "revise" for just because I think these journals are a cut above. I think that would be a disservice to the author - if they made it as far as the "under review" stage, they deserve a fair shake.

[grymes]

I don't tailor my reviews to journals. The editors' job is to decide whether the paper is a good fit for their journal (where fit includes considerations of selectiveness as well as considerations of style/subject-matter/narrowness/etc). My job is just to tell the editor what I think of the paper. What I think of the paper does not depend on what journal I happen to be refereeing for. (An exception: if the editor's referee request asks me a specific question regarding fit, I happily answer that question. Otherwise, I don't presume that the editor cares about my opinion re: fit (and I don't particularly think they should).)

[anon]

I'm with postduck, although my sense is we're not in the majority (in philosophy, anyways). This sort of decision - how suitable the article is *for the journal* rather than just *as an article in a subfield* - is a decision that should go to the editor rather than the referee.

Maybe the referee can speak to this in their report, if they want to - raise a question about whether the article fits properly within the scope of the journal, that is. But I have the feeling that instead of doing this explicitly, referees who think this is part of their job tend to just ratchet up their standards in a vague sort of way ("Is this *really* brilliant enough to warrant publication in X?").

[Marcus Arvan]

Thanks Douglas, it's good to hear at least someone so far agrees with me on this. ;) I'm actually surprised that several commenters so far think that these decisions (about significance) should be made by editors--and while I appreciate the pushback, I'm still inclined to disagree. Here's why.

First, the process of peer-review is one where both editors and reviewers constitute 'peers.' It's a distributed process of peer-review within which several different groups of peers get to have a say over whether the paper should be published in that journal.

Second, one of the primary things we do as professional peers is evaluate the significance of arguments. It's easy to put together sound arguments for trivial conclusions. It's hard to put together sound arguments for controversial and significant conclusions. So, when I see a paper that makes a tiny move in a well-worn debate, I'm apt to think: this isn't as important to publish as a paper that makes a more important, controversial, and original move (and does so well). And, since I'm a peer reviewer, I think it is entirely legitimate to make a call like this.

Third, I think the above responses imply a strange conception of the role that desk-review (by editors) plays or should play in peer review. By my lights, all that passing desk-review conveys to me is that the paper has passed some minimum threshold of competence and *potential* fit for the journal.

When the editors send me a paper as a peer-reviewer, they haven't told me or even clearly implied in any way, 'We judged at the desk-review stage' that this is a good fit for our journal. They have conveyed to me, 'It meets whatever *minimum* standard we, as editors, have for sending things out for review--but now we want to know what you think.'

Anyway, here's why I think all of this matters. A number of people on this blog and elsewhere--including, importantly some editors at the top journals in question here!--have commented in recent years about how all too many papers are 'too safe' and really not all that significant, etc. See e.g. https://philosopherscocoon.typepad.com/blog/2012/10/safe-and-a-little-boring.html

To the extent that we serve as peer-reviewers in this entire process of peer-review, I think we need to take some responsibility for this. It's not (merely) just the job of editors to determine whether a given piece is 'truly significant' enough to publish in a highly visible and venerated journal. It's *our* job to do that together. It's our job to tell editors what we think, and it's their job to decide whether we've made a good case for the verdict we recommend.

A final, potentially relevant side-note: when I do demur as a referee on the significance of a paper, I'm always very clear to the editors and author about what my concerns are, ultimately leaving it for the editors to judge for themselves. For example, I may say things to the effect of, 'This is a well-argued paper, and if the editors decide to publish it, I wouldn't argue with the decision. However, the paper seems to me to make a relatively small move in a well-worn debate, so I'm not sure that its overall significance is befitting of this journal. Here's some more detail as to why...'.

This seems to me an appropriate thing for a referee to do, though I imagine some of my peers might disagree. But hey, that's why we call it peer-review, right? As peers, we can reasonably disagree about what exactly our role should be as peer-reviewers, no? Significance matters to some of us more than others (just as 'rigor' matters to some of us more than others).

[Marcus Arvan]

postduck: I’m no fan of rankings myself—but I think the best way to address hierarchies like these are on the author’s side of things: namely, to not just submit to and cite things in ‘top journals.’ As an author, I go out of my way to cite and amplify things things that I think are significant wherever they appear, and often, I think really significant stuff is published outside of the ‘top echelon’ of journals. I just see my role as author and reviewer as playing significantly different functions.

[postduck]

Hey Marcus, thanks for your response! I agree with your point about commenting on tiny moves in well-worn debates. I would just suggest that you make that comment for any journal, not just for the tippy-top ones. And I agree with your point about peer-review being a distributed process, at least in principle. But from a practical standpoint journal editors have way more say about what topics are suitable for a journal than reviewers do - e.g. at the desk reject stage - whether reviewers think it's their job or not.

But I don't think I've got a particularly strange conception of the desk rejection stage. Maybe this varies from journal to journal, but I've gotten desk rejects from some fancy places where I've been explicitly informed by the editor that my topic isn't a good fit. Other journals make this explicit in their submission guidelines. Here's what MIND says about desk rejects:

"New submissions are sent by the Managing Editor to a member of the editorial team who makes an initial assessment about suitability. To reduce the time that authors wait for decisions, many submissions will be rejected at this stage without comments. Even highly creditable submissions will be rejected at this stage."

Obviously undergrad papers will get caught at this stage, as well as papers from random internet cranks. But so will papers that don't appear to be within the scope of the journal. And they can change what that scope is at their discretion - e.g. when MIND made a point of announcing that it was going to start considering articles on a broader range of topics: https://dailynous.com/2016/04/19/prominent-philosophy-journal-broadens-scope/

Anecdotally, I've had papers get desk-rejected that I have to assume weren't filtered out solely because they failed to meet minimum quality standards - namely, because the same manuscripts went on to get really positive reviews and ultimately acceptances from other very good journals. I usually infer it's because I work on a set of topics that some editors consider to be niche and interdisciplinary, and so my papers are not MIND or Phil Review's cup of tea. I'm sure many others have had similar experiences.

Anyway, your perspective on the role of reviewers makes a lot of sense, and I'm sure many very thorough reviewers adopt it as well. But I don't, and I think there are pretty good reasons not to start.

[Postdoc]

I have a follow-up question for those who tailor their referee reports to journals, or who received tailored referee reports.

Wiley and Springer now offer "transfer services". When you transfer your paper and referee reports to another journal, are they sometimes sensitive to the fact that a report was written for a "top journal"? I just got referee reports that basically said "That's a fine paper, but just aim lower". For instance, can you send your paper to another journal and be like: "Hi, as you can see, two referees thought that my paper was fine. However, they thought that I "aimed too high". Would you like to publish my paper?"

[grymes]

Marcus,

Just to echo postduck: I don't get why your three points are supposed to constitute an argument for peer reviewers tailoring their reviews. 1) The process distributed, as you say--the referee's job within that distribution of labor is to pronounce on the quality of the paper. 2) Of course pronouncing on the quality of the paper includes pronouncing on how significant it is! But one can easily do that without worrying about how significant it needs to be in order to fit in that journal. And 3) the editor makes decisions after getting reviews, as well as before.

With all that said, I'm not opposed to you or others making suggestions about fit in ref reports--it seems like a fine thing to do, though I'm not inclined to do it without being asked. But I'm also fine with editors ignoring those suggestions, even when they're reasonable.

[Marcus Arvan]

Hey grymes: To clarify, I wasn't offering an argument that peer reviewers *ought* to tailor their reviews to journals. As I noted in the OP, I was only reporting what I do as a reviewer and my reasons for doing it. My reply comment in this thread was merely aiming to defend the legitimacy (i.e. permissibility) of this approach against the criticism that these things should be left to editors.

Personally, my position is that (within reason), reviewers should be free to use their own standards for evaluating papers for publication. That, I think, is what makes it a system of peer-review. As peers, we all have our own unique views about what constitutes good philosophical work worth publishing in this venue or that.

[grymes]

Thanks for the reply, Marcus. Happy to grant that it’s permissible! Our two (minor) disagreements, I think. 1) to me, what makes it peer-review is just that the reviewers who have vouched for the quality of published manuscripts are peers of the authors; that doesn’t seem to me to require giving reviewers say w/r/t fit. 2) I guess, contra your last sentence, I don’t really have views about what’s worth publishing in one venue rather than another (putting aside obvious demarcations like the length of Analysis articles or the fact that Philosophy of Science articles should have something to do wirh science). I just recommend that whoever’s asking for my opinion publish the good stuff and send the bad stuff back to be reworked.

[Michel]

Hmm. Do you all mark a difference between generalist vs. specialist journals?

I mean, do you tend to think that a paper submitted to a generalist journal, especially if it's one that doesn't often feature work in that subfield, ought to be of somewhat broader interest/import, or perhaps be somewhat less epicyclical/in the weeds?

I think so, but perhaps I'm wrong to do so. When it's been relevant I've just flagged it as a question for the editor (does the journal have enough of an audience for this sort of thing) and gone on with evaluating the paper strictly on its merits.

[Peter]

I think there's a middle ground between people who think that referees ought to tailor their reports to the "prestige" of the journal, and those who think that it is up to the editors to decide whether a paper has the right scope.

When it's a paper with a narrow scope, what I tend to do is just point this out in the report: "Hey Editors, this is a fine paper that could be accepted with some work, but it's basically a reply piece -- given that it's basically a reply piece, I leave the decision up to you as to whether to allow resubmission or to reject it."

Editors don't have time to actually read everything that crosses the desk. Lots of papers that turn out to only make small moves in the literature might have, on a quick editorial skim, looked like they were major moves; why can't referees flag this to the editors and leave it up to them to decide?

[Tim]

I don't tailor my reports to the journal and I don't think we should. I don't object because of a division of labor between editors and reviewers (which is what people seem to be discussing now). I object that our intuitions about fit between journals is probably highly unreliable. And I don't think we should use highly unreliable intuitions when evaluating papers. So I do periodically have intuitions that it is surprising that (say) MIND published such-and-such paper? Sure. Do I think I should give such intuitions weight when evaluating a paper? No. I think they have little to no evidential force.

Some will point out that this policy will mean that some top journals may publish papers that should have been published in a "lesser" journal. But so what? This policy will make someone--probably a graduate student or early career scholar--happy and fortunate. And that's not such a bad outcome.

[Evan]

Several questions to consider: Are editors in a position to always or tend to know what’s better or best for the journal compared to referees given such vague and broad mission statements of many of these journals in the first place? If not, then how come? How often do editors choices contradict journals’ missions? Should we take the prestige or intellectual reliability of a journal seriously if such inconsistency occurs? Should we be worried if there are significant inconsistencies? Should referees who want to discuss an article’s fit mention or remind editors of the journal’s mission? Will doing so keep editors intellectually humble about their own biases and/or taste? If a journal’s mission is very specific, should referees still go against the grain and give their opinions about what *should* be published in that journal instead?

[Jamin Asay]

I don't think I tailor my reports to the journal. (Though I'm not blind as to the journal I'm reviewing for, so who really knows.) I can say that after refereeing for a decade now, I've very rarely had the thought that I would have given a different verdict had the paper been submitted to a different journal. But I'll make two points.

First, because it's so difficult to get published at all in any of the "top journals" (meaning, let's say, the top 25-30 in the latest Leiter ranking, plus all the well-known specialist journals), I don't think it would be warranted to be tailoring one's judgments within that group. Standards are super high throughout that group, so I'd be disinclined to be making anything but super coarse judgments about the quality of the venue.

Second, from the perspective of a reader, publication venue doesn't really matter. If there's an argument I need to address in a paper I'm writing, it doesn't matter if it's in Ratio or Nous. It needs to be addressed. Post-publication, my judgment of whether a paper is significant and worth engaging doesn't turn on the venue; it turns on what the paper says. I'm going to cite it and engage it regardless of where it's published. So, pre-publication, I don't really care where it gets published either.

[rookie reviewer]

Hi everyone, OP here. Just wanted to say that it's been super helpful to read your thoughts on all this—thanks for taking the time to weigh in!