Our books






Become a Fan

« Real jobs in philosophy part 5: Leigh M. Johnson (Christian Brothers University) | Main | Cocoon Party/Meetup at the Pacific? »

03/28/2016

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

recent grad

Do you have any details on what "graduate student writing" tends to consist of?

Stacey Goguen

Also, some research has suggested that people expecting to be held accountable for their decisions and judgments may exhibit less bias in those judgments:

"Among those [approaches to debiasing] for which research evidence suggests the possibility of successful debiasing outcomes include:[...] Having a sense of accountability, that is, “the implicit or explicit expectation that one may be called on to justify one’s beliefs, feelings, and actions to others,”
can decrease the influence of bias (T. K. Green & Kalev, 2008; Kang, et al., 2012; Lerner & Tetlock, 1999, p. 255; Reskin, 2000, 2005)."
(Source: http://kirwaninstitute.osu.edu/wp-content/uploads/2014/03/2014-implicit-bias.pdf)

So there's reason to think that reviewers' judgments may improve if we consistently expect them to provide justification for them.

Wesley Buckwalter

I saw two approaches being advocated in the comments, one to write "a couple of lines" the other to write "1500-2000 words". I would advocate for something in between. In my view, the main job of a reviewer is to evaluate whether a paper has met the minimal standard unit for publication. So, in most cases of rejection, I think the reviewer should clearly articulate the main ways that the paper failed to achieve that unit. But, this need not and probably should not take "2-6 pages" to communicate! That many pages begins to seem like clandestine co-authorship and it strays from the main job of the reviewer. It might also be less optimal for the system on the whole, if fewer people accept assignments or complete them punctually because of extensive commenting.

Is there any evidence that making people justify their responses reduces rejections as a result of the biases you list? It could lead people rethink their reactions, but it seems more likely that they just look for reasons to justify them. You can find reasons to sink any paper if you really wanted to, and often any reason is all it takes to get rejected. In any event, if the reviewer is going to recommend rejection, I'd rather just have it be a few sentences and quick so I can move on elsewhere, rather than wait around for someone to write 2000 words about my submission, before I go ahead and do that anyway.

Hugh Miller

"Out to Pasture" (... "Lunch" ?) does not seem to understand the concept of due process, or think it applicable in refereeing. Due process requires that one not only decide, but give (more or less) public reasons for one's decision. The strength of the argument they assert may be contestable; but the argument must be made.

Postdoc

'Here's a brusque (but I hope not brutal) headline thought. Referee's reports for journals are written for editors. They should offer a judgement about whether the paper is worth publishing. They are not, primarily, tutorial reports for authors. If the judgement is negative, then it is a work of supererogation to spell out the reason in any detail: a couple of lines for the editor's eye is enough. The editor should trust your judgement if s/he has chosen you to report. It is nice if you have the time to say more, but there is no need.'

In reply, I have just two things to say.

First, I have no problem with this, as long as it's applied to rejections and acceptances. I had a paper basically accepted at a decent journal and the editor ignored the verdict, because the referee had not written enough justifying it. On the other hand, I've had papers rejected for little reason. Sometimes, I am provided with no reason whatsoever, so I can only assume there was no reason, or no reason they dare share.

Second, some top journals say 'We cannot provide comments on all rejected papers. We focus rather on arriving at a well-informed judgment without undue delay.'

But there seems to be an inconsistency between ‘arriving at an informed decision’ and ‘not providing comments.’ If your decision is informed, then you have comments and can provide them. So, I can only surmise that this policy is designed so they can reject papers for bad reasons to keep their rejection rates high and continue to be perceived as prestigious.

That's all

Marcus Arvan

recent grad: That's a great question! I hope to write a post on it in the next couple of days. As I explained in a post a while back, "On not acting like a grad student" (http://philosopherscocoon.typepad.com/blog/2014/03/on-writing-and-acting-like-a-professional-rather-than-a-grad-student.html ), my sense is that there are two general qualities that people associate with grad students: (1) underconfidence, and (2) overconfidence. My own sense, in learning how to publish, is that indeed, one must learn to avoid these pitfalls as a writer as well. In my upcoming post, I will try to further explain my sense (which could be wrong!) as it pertains to writing specifically. In any case, thanks for asking: it's a great question!

Marcus Arvan

Hi Wesley: I think Stacey put it well.

First, accountability can plausibly incentivize people being more careful. A person may think a paper is bad to begin with, but when they sit down and actually have to compose a *detailed* review, they have to provide an argument for their recommendation--and they may find that their argument is difficult to make, which could in turn get them to reconsider their recommendation. This has happened to me before as a reviewer. Sometimes I initially have a negative opinion of a paper, but when I find myself having to *explain* my initial reaction, I have to be more critical of my own reactions--and better understand the paper.

Second, even if justifying a recommendation doesn't make a reviewer more careful, it can reduce bias by subjecting that evaluation to a second set of eyes: the editor(s), who can then *compare* that review with other reviews from other referees, to see if it is actually a competently performed/unbiased review. In some cases, I appear to have editors disregard a middling or even negative review because (A) the other reviews were very positive and (B) the negative review gave shoddy justifications for rejection that the other reviews undercut. So, this is a second way actual justifications can plausibly lead to better results. They give editors actual justifications to judge for themselves, along with other reviews. Failure to give any information at all (besides "trust me") deprives editors of important information.

Wesley Buckwalter

Hey Marcus, I am familiar with some of that research, but I'm not sure how it would extend to this particular environment or set of questions. I'm also not sure I see how much more or any "accountability" there is for an anonymous reviewer of a 1000 word report than one who did a 100 word report. How are anonymous reviewers being held accountable in either case exactly?

Scott Clifton

"Second, some top journals say 'We cannot provide comments on all rejected papers. We focus rather on arriving at a well-informed judgment without undue delay.'"

Yep, and I love it when I get both a rejection without comments eight weeks after submission. Really? It takes a reviewer that long to give an up or down vote? I guess they don't consider two months to be "undue delay."

Helen De Cruz

I get a lot of referee requests. I accept what I believe is a fair number (maybe about 10-12 a year). Lately I have revised my refereeing practice to be less micromanaging and briefer, also to be more efficient with my time. I do not think reports of 3000 words for an 8000 word paper are ultimately helpful for the author, and they are a huge time sink for me.
So I currently write referee reports that are about a page, and never more than 2 pages (about 500-1000 words) long, also if I recommend revisions. They are shorter if I recommend rejection, especially if the paper is of very poor quality. I begin by saying what I think is good about the paper, and then briefly review the worries I have about it. I try to balance being honest and being useful: if a paper is poorly written, the author has to know this or else they would be sending around a paper forever that is rejected on those grounds. If it misses discussion of key portions of the literature, I give a few examples but I do not think it is my task to help the author to all the sources they miss. If there are flaws in the argument, I point them out. I try to restrict myself to no more than 3 big comments and a couple of smaller comments.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)

Job-market reporting thread

Current Job-Market Discussion Thread

Categories