Our books






Become a Fan

« Prospective-student visits and PhD advisors | Main | Spring break sounds »

03/14/2019

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Trying Not to be Cynical

I haven't encountered this often myself. I believe I've received at least one set of referee comments for just about all of my (non-desk) rejections, even from top tier journals.

I know I've heard various philosophers admit, through blog comments, that they only provide a sentence or two of cursory comments (written in a tone clearly intended for the editor) when they feel the paper obviously should not be published. It's easy to imagine that editors would not pass on comments which simply read "The thesis is unclear and the argument is obviously bad; reject". I'm less sure of it, but I thought I've heard, through the philosophy blogosphere at least, that some journal editors sometimes solicit referees on the understanding that such cursory reports are acceptable. Part of the justification (as I vaguely recall) is that you can get faster referee turnaround if you don't ask for substantive comments and only want a recommendation.

Nicolas Delon

I don't have insight, but something I've read that sounds plausible and that echoes 'Trying Not to be Cynical's comment, is that no comments means the comments were terse and negative yet sufficient for making an informed decision. If an expert tells me in two sentences that a paper is poorly argued and doesn't demonstrate adequate knowledge of the field, and say those things to me in not such friendly terms, that may be a sufficient ground for rejection and yet I may not want to share those comments with the author. In fact, long wait plus no comments is (arguably defeasible) evidence that your paper was no fit for the journal or ready for submission.

I'm not saying I endorse the practice and I've almost always received comments on my submission except when they were desk-rejected, but I can imagine why editors need not tell authors that someone literally thought their paper was just terrible. For let's be honest, there are many terrible papers being submitted and oftentimes that's the only thing you can say: it's terrible (poorly argued/informed/written...).

Michel

This has only happened to me once, so far, that I remember. It was at a middling subfield-specific journal that's known to be somewhat idiosyncratic (in ways I'm not yet privy to).

Like TNtbC, I think it's probably just because the editors tell their referees that a sentence or two justifying their impressions will suffice, and it's not worth passing that on. I find it hard to accept that 3+ months is an acceptable turnaround time in such cases, though.

Another possibility, of course, is just that the editor deemed that the comments weren't worth passing on for whatever reason--maybe they're too nasty, for example. But I doubt that accounts for most, let alone all, the cases.

Marcus Arvan

Nicolas: I suspect you are probably right about the rationale - but I am unconvinced the rationale is a good one.

Peer-reviewers often appear to get things dead wrong:

https://s3.wp.wsu.edu/uploads/sites/285/2015/08/GansAndShepard.pdf

https://philosopherscocoon.typepad.com/blog/2015/09/stanley-on-peer-review.html

I also balk at this: "For let's be honest, there are many terrible papers being submitted and oftentimes that's the only thing you can say: it's terrible (poorly argued/informed/written...)."

I've reviewed papers I thought were terrible. However, in every case, I gave a detailed explanation for precisely why I thought they were unpublishable. Bias being what it is (human beings are often terrible at detecting their own biases), I think the proof is in a pudding: if a paper is bad, you should be able to succinctly *show* why it is bad, not simply say that it is. Further, I have first-hand experience for why this is a good practice. In one case, I submitted a review explaining why I thought a paper was bad--and because two other reviewers thought the paper was not so bad, the author was given a R&R and ultimately convinced me in their author's reply that I was mistaken.

In my view, this kind of transparency--and indeed, accountability--is vital for a peer-review process to function properly.

Marcus Arvan

I would also suggest that, practically speaking, forwarding reviewer comments may be a much better way of deterring authors from submitting papers that referees deem 'not ready' for submission (which I have heard many an editor say is a serious problem these days!).

When an author receives no comments, I expect they are liable to just send off the paper to another journal - thus 'clogging up the system' with a bad paper (if indeed it is bad). In contrast, I suspect that if an author received a set of negative reviews, they might be much more liable to reconsider whether the paper is suitable to be placed under review again.

Nicolas Delon

Oh I agree, Marcus. I'm just also trying not to be cynical and to give editors the benefit of the doubt.

But I think the burden is on referees to provide justifications for rejecting terrible papers only when this can be done succinctly. If not, then the burden is on the author to prove, through a better paper, why they are entitled to feedback. Sure, there's disagreement about whether some papers are terrible or not, but I shouldn't have to waste time crafting thoughtful comments for papers that fail the standards of even a decent paper at every single page. And if authors end up 'clogging up the system' by re-submitting their paper elsewhere without changes, it is again not referees' fault.

Perhaps a good proxy would be: if after X months you receive a rejection without comments, have someone take a fresh look at your paper before resubmitting it. Silence is probably a sign that too much was wrong for comments to be worth writing. Of course, more people would need to understand the implicature, which would have to be transparent enough to do its job.

All that said, I totally agree that referee reports sometimes, if not often, fail to provide necessary information, and that's bad practice. But as we all know, referees are also time-constrained and doing unpaid work.

Marcus Arvan

Hi Nicolas: I think we're largely on the same page - though I'm inclined to think if a paper is *that* bad (failing the standards of decency on nearly every page), it should be pretty easy to type up why in like 10-20 minutes.

Even if the author isn't entitled to it, it may nevertheless be a good practice for referees to follow--if only to show an author that they are submitting work that is nowhere near close to adequate (which again, I think might help deter authors from clogging everything up with such bad submissions - which would be good for everyone, including referees!)

Douglas W. Portmore

"What does seem more puzzling is when a paper has been under review for a number of months (2-3+ months) but no comments at all are forwarded to the author. Assuming the papers did go out to referees, there seem to be only two possibilities: either the referees gave no comments, or the editors did not forward the comments to the author."

I don't think that you should assume that if a decision doesn't come back until 2-3 months, then it must have gone out to external reviewers. Some top tier journals (e.g., Philosophical Review) start by reviewing the paper internally. Given that this involves such a small number of internal reviewers (in the case of some journals, I believe this rotates among various two-person teams) looking at every submission that comes in, it's not unreasonable for it to take 2-3 months for a desk reject. Also, there is a third possibility that you overlook: it was sent out to reviewers who provided comments but the reviewers provided no comments for the author. Providing comments that are helpful to the editor in reaching a decision about whether to accept a paper can differ significantly from providing comments that are helpful to the author to understand both why the paper was rejected and what they might do to improve it. Because of this, some journals don't insist that reviewers provide comments of the latter sort.

The problem is that providing authors with a timely response that helps them understand both why their papers were rejected and how they might improve them doesn't necessarily help a journal achieve maximal prestige in terms of the influence of the work that it publishes. And since many authors seem to be more concerned with the prestige of a journal (in terms of whether it has a reputation for publishing influential work) than they are with whether it has a reputation for providing authors with timely and helpful feedback, many journals achieve maximal prestige despite having a bad reputation with respect to providing authors with timely and helpful feedback. Thus, it shouldn't be puzzling or surprising that some journals might be interested in achieving greater prestige (and, thus, better submissions) at the cost of failing to provide authors with timely and helpful feedback. After all, search committees and promotion committees only care about the reputation that the venues that you published in have for publishing excellent and influential work. They don't care at all whether the venues that you published in have a good reputation for being helpful to authors whose work is rejected. Thus, authors have an incentive to submit their work to such venues. And this allows them to get better submissions, which helps them get better submissions still. Thus, the incentive structure is both for journal and for authors to be more concerned with the prestige of the journal than with their reputation for being helpful to authors whose papers are rejected.

Really?

Anecdotally, I have heard that the majority of rejections at PPR, Nous, and Phil Imprint do not receive comments. (I have some personal experience here too.) If the current explanation is correct, this implies that the vast majority of submissions to these journals are crap, and worthy of a 2-3 sentence rejection. Surely top journals do get submissions that are terrible, but are we to believe that *most* are *that* bad? There is also an assumption being made here, viz. that the 2-3 sentence justification will always be unhelpful for the author. This might be false.

Anyway, one thing I have no sympathy for is the idea that passing on the comments adds an onerous layer to the work-load which would materially increase wait times. This is what the quote from journal, given by the original poster, suggested.

Amanda

I have had this happen with papers that, without changes, were later published in top places. So I don't by the whole "the paper is just too bad to say anything." Sometimes I suspect it does take months to desk reject. But if there are comments, I do think they should be sent to the author, even mean ones. That just seems honest.

Grad Student

I have had three rejections without comment (from the same two top-five generalist journals) which, given the timeframe, clearly were not desk rejections. The papers rejected were not unfit for publication — two of them have since been accepted, one in a comparable generalist journal and the other in the top specialist journal in its area. I have also spoken to a number of other philosophers whose papers have been rejected from the same two journals without comment after months under review and then accepted at comparable journals. Unfortunately, I believe the best explanation for this phenomenon is that the journals are allowing or encouraging referees to come to hasty and poorly considered verdicts.

Trying Not to be Cynical

Nicolas and Marcus, I think a relevant point is that some referees are actually bad at their jobs. I like how Michael Huemer puts it on his website: they answer the question "Is this paper correct?" when they should be answering the question "Does this paper contribute to the literature?" or "Will other people want to read this paper?".

The point is that when a referee essentially says (via their cursory report) that a paper is so obviously bad that it's not worth comment, they often really mean that they think it's so obviously wrong as to not be worth commenting. Huemer uses the example that many ethics papers are rejected by Kantian-leaning referees because the paper presupposes utilitarianism (and therefore that referee feels it's "obviously wrong"), or vice versa.

When referees don't write substantive reports, it's hard to tell if this sort of malpractice is happening. And, I suspect that cursory referee reports hide this sort of malpractice just as often as they indicate the submission was genuinely unpublishable (e.g., by lacking any argument at all, by failing to engage with the relevant literature, by equivocating concepts well understood in the area, or by lacking a coherent original thesis). At the very least, the malpractice case isn't behind a negligible percentage of cursory reports.

A different grad student to the one posting before

I wonder whether one solution to this might be to push for journals to have a form reviewers could fill out along with their review - that way, if their comments are scant (as a few commenters suggested) it might at least indicate some problem areas to the authors.

abd

Relatedly, can one reliably tell that a submitted paper has been sent out to reviewers when a journal submission site labels the paper as "in review"? Or do papers that have been labeled "in review" sometimes end up being desk-rejected? Perhaps the variation in submission sites, together with the practice of rejecting papers without passing on reviewer comments, makes this too difficult to answer helpfully.

Frustrated Senior Philosopher

I have a paper that's been to four journals now, under review at each for three or four months, without ever receiving comments. I think it's one of the best things I've ever done (it's a historical paper, on a figure on whom I've published widely and well, and I think it's the best thing I've ever written on that figure). Maybe it's unspeakably awful -- but I've presented it successfully at conferences, even used it as a job talk (and was offered the job). So I don't get what's going on, and it's very frustrating. I'm fairly senior and certainly not desperate to publish, so I can only imagine how a junior person under time and career pressures must feel about this kind of malpractice. And it *is* malpractice.

Pendaran Roberts

I don't understand the referee process at many top journals. The process is incredibly opaque, and given that they often provide no explanations or justifications for their decisions, way too slow. These days due to the horrible job market people feel pressure to submit to the fanciest places they can, but really many of these journals are not designed for early career philosophers.

My favorite top journal is Synthese. They're not as elite as PPR or NOUS but they're considered roughly a top 10 journal (depending on who you ask), and they always (at least for me) return at least 2 sets of referee comments, usually within 3-4 months. The editors respond to emails quickly and treat authors with a lot of respect. They're a very professionally run top journal, based on my experience. I submit most of my work to Synthese first, as it's such a pleasure working with them.

abd

I'm waiting to hear back on a paper that's marked as "in review" right now, and had been feeling good that the paper hadn't been desk-rejected until I saw this thread! Is it your experience that some papers that move from "with editor" to "in review" still end up being desk rejected?

Amanda

Abd:yes

Douglas W. Portmore

There can be different levels of desk reject. For instance, at some journals the Editor-in-Chief will desk reject a certain proportion. Then those not desk rejected by the Editor-in-Chief will go to an Associate Editor who will desk reject some of them and send others out to external reviewers. Also, I think that some journals might use the "with editor" designation just to mean that it's being processed by the managerial editor and "in review" designation to mean that it's being reviewed by an Editor, Associate Editor, or external reviewers. So, I think that you cannot infer from the fact that your submission has moved from "with editor" to "in review" that it won't end up being desk rejected. Many managerial systems are not well tailored to our discipline and especially to journals that want to practice triple-blind review. As a result, the designations in the managerial system can often be misleading.

Craig

I'm very sympathetic to the request for comments. However, I want to offer a counterpoint. A substantial proportion of the papers I receive to review are very disorganized, to the point where the argument structure is unclear, the thesis is unclear, and the engagement with the extant literature is unclear. When I read them, I think, "*This* point, and *that* example are interesting and provocative, but I'm not sure I understand the paper." Perhaps explaining why a paper omits to engage a potential objection or an extant bit of the literature only takes 10-20 minutes, and perhaps explaining why a stated conclusion is not controversial or interesting only takes 10-20 minutes. However, explaining why a poorly written paper is poorly written takes much longer than 10-20 minutes (for me, at least) -- especially if I think, "Maybe I've just misunderstood this; I should set this aside and read it again later."

Accordingly, there are times where I am very confident very quickly that I will *at best* recommend revise-and-resubmit-with-_major_-revisions, and yet I realize that, given the other demands on my time, I will not be able to provide more-than-conclusory explanations for that effective rejection any time soon. My inclination has been to hold on to the paper until I have time to sit down and write something more substantive, but that can take quite some time (weeks, months).

Maybe I am an inefficient referee, and I should just not accept requests to referee? That seems the wrong result. Maybe I should become a more efficient referee? Sure, but that will take me a much longer time, maybe a career, and I have to decide what to do in the meantime. But as long as I'm going to referee, and as long as a substantial portion (and perhaps a majority) of the papers I receive to referee are disorganized and poorly structured, I find myself facing a very steep tradeoff between prompt reports and explanatory reports. Thoughts?

Nicolas Delon

The original question is why journals do this, not whether they should. Is the preferred explanation that they just don’t care? That sounds neither plausible nor charitable. Surely, if they do this so frequently (assuming they do, and I’d like to see evidence), there must be a better explanation than referees/editors don’t care. After all, these are the same people who submit their own articles. We can’t dismiss the hypothesis that they do reject papers without comments because those comments provide sufficient information for editors that authors need not know about just because we think they shouldn’t. These are separate questions. As to whether they should, I don’t know. Sure, many papers probably deserve more comments than they get, and that’s a real pity, but many more papers need improving before being entitled to comments. Rejection is information—not great but information still in a very nonideal system.

As for PPR and Nous, with which I’ve got very little experience, I’ve read that they first send to one reviewer (or associate editor?) papers that are not desk rejected for a quick check to know if it’s worth sending for actual review. Desk rejected papers don’t get comments (that’s kinda the point), but others do I believe? I might be wrong.

Paul

I don't have much experience with top journals, but I have only received one desk rejection without comments, all others came with at least a paragraph, most with more. I will reiterate what has been said here before, that it, that specialist journals give the best feedback and are the most responsive in my experience. So, unless you think you have to, or just have an article that you think is the bee's knees, then I would just start with the top specialist journal in your area. Saves a LOT of frustration in my experience.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)

Job-market reporting thread

Current Job-Market Discussion Thread

Writing Service


Categories