Our books






Become a Fan

« Reader query on posting drafts online | Main | Query on length & structure of teaching dossiers »

08/20/2018

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Michel

I've only been pretty sure of an author's identity once so far (that guess bore out, in the end). It would have been hard not to guess correctly, though, given the topic.

I, too, mostly target journals with good turnaround times, although that list is limited somewhat in that they have to be journals that at lest occasionally publish in my subfield. And I've generally been pretty happy with the quality of the comments I've received, with the exception of one paper (which took years to place).

It's in a fantastic journal now, but I'd just about given up on it, and only sent it there on a whim. Although the first few rejections were useful, I was exhausted and discouraged by years of subsequent reviews telling me it was unpublishable because it wasn't philosophy (even from a journal claiming to want unusual and controversial material!), or simply not engaging with the paper at all. They were wrong: it's a good paper and a piece of original research. But holy crap, was it hard to get anyone to take it seriously.

I strongly suspect that, in the end, what did it was that I decided the paper needed a cover letter saying that I already know it's an unusual piece of work, but explaining why I think it's a valuable contribution, and why the unusual aspects are necessary to my argument.


More recently, however, I've experienced (for the first time) a gate-keeping phenomenon I'd only ever read others complaining about. I submitted a piece of historical work on an entirely unexplored topic, and the reviews, while positive, said that I needed to cite works V, W, X, Y, and Z. I knew of X, but decided against citing it because I think it's a very poor piece of work that's only mildly related to my topic (but I accept that this may have been the wrong call). When I looked at Y and Z, I discovered that they're not at all relevant. Like, not *at all*. And V and W are relatively obscure works in a language I don't really know; that's entirely my problem, but given X, Y, and Z, I rather suspect V and W won't prove to be especially useful either. I'll end up citing them all, because needs must, but I think doing so is a silly exercise. And I think that if I worked in history, and encountered this more regularly, I'd find it seriously discouraging.


Finally, on scooping: I think I've been scooped a couple of times. But the extent of the scooping was pretty minimal, and hasn't proven any kind of obstacle to getting my own work out there. And I'd say *I* scooped someone once, but it looks to me like the same is true there: it hasn't proven a problem. Instead, it looks to have started a small cottage industry for us.

Amanda

I think the peer review system needs to change, but I'm not sure how. I think trying out a number of different possibilities, including the one Marcus advocates for, would be nice.

When peer review works, it works great! And I would guess it "works" maybe 40-60% of the time. These are the cases in which it is actually blind, and the reviewer does a competent and fair job.

Here are what I see as the problems:

1.A significant percentage of the time it isn't blind. A friend of mine on fb recently posted asking about what to do if you get a paper from someone you know. *Many* people openly said that they review the paper anyway, justifying this by the following claims, (1) they believe they can be unbiased; (2) It is inevitable, and (3) some said they checked with the editor first. In any case, the problem I have with this is it completely advantages people already in cliches, the evidence shows we cannot know when we are biased, and calling the process blind review when a large percentage of the time it isn't is just, well, dishonest.

2.Publishing favors boring work in every field, e.g., too much time is spent on literature review to make a tiny point at the end. Outside of favoring boring work, many top journals favor M and E, even though this is a field which has less and less job openings every year. So it makes philosophy more and more of a field where the few people at elite institutions who work in MandE publish in top journals, and other people working in fields better suited to the job market have a more difficult time of it. (I should note a number of journals have expanded their mission lately, so perhaps we are moving in the right direction here.)

3. Philosophers have terrible manners. I work in psychology and other disciplines sometimes, and have many friends in other disciplines They absolutely cannot believe just how plain mean philosophers are in reviews and talks. And then what kills me is that someone will stand up all self-righteous and insists that the cause of "reason" requires them to be an ass. No, not really, consider the following two statements:
(1) This was an uncreative piece of work, repetitive, and has nothing new
to offer to the literature.
(2) This paper needs to be improved by adding more material that will
advance the literature in a new direction, and more creative insights to
keep the reader interested.
The above statements say more or less the same thing, but the first one is someone just being plain mean.

3. Having both maximum and minimum word limits is not helpful to anyone other than the publishers.

4. I can't see any reason not to have triple blind review, other than this would stop editors from playing favorites, which is really a reason in favor of it.

5. And yes, many referees do not read papers carefully, to put it mildly. But I can understand why: there is no incentive. We need an incentive, I like the idea of paying referees but I can't see that happening, unfortunately. So I guess we would have to do something else...

Peter

Hi Michel,

Thanks for the comments. Your experience with the unusual piece is interesting. I have never written anything substantive in the cover letters that I attach, but I can see where your approach could be a good one.

The gate keeping phenomenon you mention is disheartening. Were the papers you were asked to include from major players in the field, or represent some particular method or clique?

Peter

Hi Amanda,

Thanks for the comments. I am curious about one of the claims your friends make. One of the responses you mention that is used to justify refereeing a paper from a known author is that it is inevitable. I take it the idea is that some papers are such that pretty much every qualified referee will know who the author is. Perhaps this is because the paper is clearly an extension of a particular author's work or is written in a particularly idiosyncratic way. In other cases the work may have been made so public in either conferences or online that everyone in the sub-field is already familiar with it.

Now, I wonder how common this sort of thing really is. However common it is, it will clearly be more common in the cases of papers from famous people. If we assume (which seems reasonable) that referees tend to be more positive than they otherwise would be when they know that the author is a particular famous person, then it does seem that we have a problem. In these cases, though, it is tough to see what an editor is to do. If an editor really cannot find any qualified referee that cannot tell who the author is, then either blind review needs to be abandoned in this case or the paper will need to be rejected out of hand. Neither seem great. I am just not sure what the solution to this problem is.

Of course, eliminating pre-publication review (by having an online archive in which people post their own papers) would level the playing field in one sense, since journal referees could always look at the archive to see who the author was. The worry, I suppose, is that non-famous people will not get a boost from non-blind review, but famous people will. This particular issue attracted some discussion in the comment thread for this post:

http://philosopherscocoon.typepad.com/blog/2018/07/changing-peer-review-as-good-a-time-as-any.html

Marcus Arvan

Hi Michel: Your experience parallels mine in many respects.

Like you, I've had some very good experiences--including experiences with wonderful reviewers who provided detailed, constructive comments that I have made my papers immeasurably better. Like you, I also try to target journals with good turnaround times (though I've found these can vary at journals with good averages).

However, my sense is also similar to Amanda's: that things work well maybe 40-60% of the time. I've endured some brutally long turnaround times (even from journals with good reported averages), and about half the time I get reviewer reports that seem to consist of 3-5 dismissive sentences (simply reporting the reviewer's judgment that the paper isn't publishable, but without giving any substantive justification for the judgment). Further, like Amanda, I've often encountered reports that just seem meanspirited (something I try very hard to avoid in my own reports, and which I think editors should not allow).

Perhaps more to the point, as I think Michel's multi-year saga with his one paper indicates (I have gone through similar sagas)--and as the piece I linked to previously on famously rejected economics papers--when things go wrong in our current system they can *really* go wrong in ways that have great costs for authors (viz. years of stress, nearly giving up on a paper, etc.).

As for Peter's question of whether the system is a lottery, this isn't something that anecdotal reports will settle. However, (1) I've heard many accomplished people give examples similar to Michel's (of papers that ended up in a top journal after dozens of rejections), (2) the economics piece I linked to gives many similar examples, and (3) as Remco Heesen and Liam Bright point out in their paper, empirical studies of interrater reliability among reviewers suggest it is hardly better than chance. All of this suggests that the system *is* a lottery in significant respects.

Marcus Arvan

Also, in terms of “scooping”, to my knowledge I’ve never had an idea stolen by someone else. However, I have been “beaten to the market” on several occasions where I had paper drafts giving argument X months or years before someone else published on X (where X is now considered a significant argument in the discipline). I may not have defended it as well as those who eventually went on to publish in it, but in an “open” peer review system my drafts would have at least been out there rather than where they are now (buried in old folders in my computer desktop).

Amanda

Hi Peter,

I found a number of things particularly interesting about my friend's post. One was that so many people were willingly to publicly say that they regularly reviewed papers when they knew the author. It didn't surprise me that this many people know the author: I'd known it happens frequently for a long time. What I didn't know was how unashamed everyone was. Another interesting thing is that a few people posted that they would never review paper if they knew the author, and they were shocked other people did. This was kind of satisfying honestly, because it would drive me crazy when sweet, honest, yet naive philosophers would tell me that the identify of the author was almost never known in peer review. Re the inevitable comments....I think what they meant was that a number of fields are fairly small, and so if an expert is to review the paper there is a high likelihood this expert has read it before in a paper exchange, seen it at a conference etc.

Marcus Arvan

Amanda: I have come across those kinds of posts as well (on fb), and it has shocked me how a fair number of people seem to think about this stuff. I too saw more than a few people say, “If I know who the author is, I just inform the editor and leave it up to them whether I should still review it.” No! This completely defeats the whole point of “anonymized” review, the thing that—as we’ve seen this blog—so many people claim to be important. The fact that some reviewers really think editors should decide whether to compromise anonymized review is a serious problem.

And this is really the crux of the case I’ve long trying to make for an “open” system of peer review. In our current system, all matters of accountability take place behind the scenes—between editors, reviewers, etc. But while there are some very good editors and reviewers, as the discussions that Amanda and I allude to illustrate, some people don’t seem to follow the very rules (viz. preserving anonymous review) that are supposed to ensure fairness. And this is what an open, public system would serve fix. It would make everything about peer review far more open, transparent, and accountable to the profession at large.

Amanda

Yeah I find it surprising just how confident people are that they won't be biased. Odd given all the evidence of this suggests otherwise. But now I find myself in a confusing position when I get a paper and I know the author. Especially if the author is not someone fancy. The problem I have is so many fancy people are already advantaged by the fact blind review isn't blind, and part of me doesn't want to make this advantage even worse by declining to review. I know this is basically a form of "two wrongs make a right argument," but alas, I still find it somewhat compelling.

Marcus Arvan

Amanda: this is how issues in our current system can compound themselves. Now that you see people apparently making exceptions for themselves in compromising anonymized review, you wonder whether you should make exceptions in your case. In which case we have a bunch of people making exceptions, and we don’t really have an anonymized system (or, at least one that ferrets out and prevents exception-making).

I just received a request to review a paper whose author I know yesterday. Although I will decline, citing my knowledge of the author’s identity, I would be lying if I said that I didn’t have the same momentary thought report (namely, why not make an exception given that many others seem to?). It is these kinds of conflicts of interest that a well designed peer review system should reliably address—and if the social media discussions you and I have witnessed are any indication, it is quite unclear whether our current system does so adequately.

Michel

Peter: The authors of X, Y, and Z are, but I don't know about the authors of V and W--I think not, but they may well be part of the circle. I'm a relative outsider, so it's hard to tell. It definitely felt like I was being asked to pay my dues, though, and I didn't much care for the feeling.

(All that said, it's a single instance, and my judgement of the situation could easily be off.)

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)

Job-market reporting thread

Current Job-Market Discussion Thread

Writing Service


Categories