Our books






Become a Fan

« Reader query on preventing 'google reviewing' | Main | Book review of Karen Kelsky's "The Professor is in" »

05/19/2017

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Stephen Bloch-Schulman

Folks,
I would agree with Marcus that I want to read and find papers that are good and worth publishing (if not immediately, then after revision). For a while I was reviewing something like 10 articles a year, which meant reading 9 or sometimes even 10 that I didn't think were publishable. That is a serious drag. I would spend a lot of time writing comments, but often, with very little chance that the article (or my hard work on the review) would ever see the light of day. So it was always a real joy to find something that I did think would be publishable (at some point). I too am looking forward to reading new exciting work, and want to read work worth publishing. If fact, I now review much less because the rejections (and R&R's that needed so much work) were a drag. I might put it this way, though I am sure I would be overselling my point a bit--reading articles is the opposite of families for Tolstoy: Bad articles are all alike; every good article is good in its own way.

Three other notes: first, I have found that I almost always know surprisingly fast that something will be rejected. Often the abstract is sufficient. I almost always read papers in this order when I review: abstract and, if it looks promising, the paper. If I am already skeptical (which I often am after reading the abstract), I immediately go to the works cited/bibliography, because when papers are not working, it is almost always evident right away in what is being used. And, again, even when I am skeptical, I am looking with hope that there might be something there to save, some promise that can be held up for revision.

Second, one of my biggest surprises as a reviewer is the weakness of revisions. I have very rarely found that an R&R becomes a publication; my untested hypothesis is this: if an author/authors couldn't do X the first time, they are unlikely to be able to do it when X is suggested from a reviewer (even if they want to). Others might have a very different experience here, but this has been my experience over and over again and so I find a tendency to reject more and leave R&R for work that is pretty close already.

Third, the way I often make a final decision is how quickly problems come up and how obvious I think they are. I am looking for people to disagree with, but people I could disagree with and feel like the argument I am disagreeing with is smart and interesting and requires a lot from me to disagree. If, as I read, I am thinking all along: what about X? Wouldn't X be obvious here? Nooooo... talk about X? Explain why X isn't the case.... " then that typically is a rejection. And I would suggest, for me, this happens a lot.

Stephen

NK

With question 3, I also had something more specific in mind, prompted by this experience: I recently had a paper (an R&R, actually) rejected, and the report involved an objection to a controversial claim my view commits me to. But the referee didn't *treat* the claim as controversial; they treated it as obviously false, backing that claim by, essentially, a bare appeal to their own intuitions. It seems to me that this is irresponsible. I would think that, if a claim that isn't one of the author's central claims is controversial, you grant it to the author for the sake of argument. Otherwise, it seems like you're assuming your own views on a related controversial matter in dismissing the author's argument, in a context in which the author doesn't have adequate space to respond.

Which is to say, here's one thing I think about reviewing (though I haven't run this past anyone before, hence my question): You should generally try to assess the author's argument on its own terms, rather than on your terms. And part of the reason that I think this is that, if I didn't review this way, it's hard to imagine that I'd ever recommend acceptance of anything I didn't write myself. The upshot, I think, is that you may need to grant authors a lot of claims, for the sake of argument, that you think are false. (Of course, there should be limits to what you accept even for the sake of argument. But my thought is that the claim should have to be pretty uncontroversially false for you to refuse to accept it for the sake of argument.)

It also seems to me that, if we don't review this way, then we force each other to do the thing Marcus recommends: anticipate every possible objection and misunderstanding. Maybe I'm more antipathetic to that than most. But it's hard to believe that you can do something interesting (and especially hard to believe you can do something interesting and systematic) if you have to write that way. (Or maybe this is what books are for?)

OH

Stephen, you write:

"...when papers are not working, it is almost always evident right away in what is being used."

I'm curious- could you elaborate on how a paper's poor quality is reflected in its bibliography?

Amanda

I have to admit Stephen, your views on reviewing worry me some. I am not sure how you could tell, so consistently, that a paper is not worthy of publication by reading the abstract. Perhaps someone is just not good at writing abstracts? That doesn't seem a good enough reason to reject it out of hand. Also, if 9 / 10 papers you read are clearly not worth publishing, I am curious what type of papers you are getting. I mean most papers (I would assume) are written by colleagues fairly good at philosophy, and that have spent a fair amount of time editing them. I would think low-balling 1/2 of papers are like this. So to think 9/10 are so bad that they "have little chance of every seeing the light of day" - well - then that says something concerning about the profession. It suggests that most professional philosophers consistently submit work that is of such low quality it has no chance of being seen in a professional journal.

In my experience, (and I have only reviewed maybe 10-15 papers ever) about 1/2-2/3 of papers at least have hope. (lower of course for the very top journals). This does not mean I give a R&R on all of those, because sometimes the hope is just so much work I think the paper needs more editing before a R&R is appropriate.

The main things I look for are clarity and originality. I think originality and innovation are way underappreciated in philosophy journals and I love that by reviewing I can help make a change for the better. I also think philosophers tend to use too much jargon and complicate matters that do not need to be complicated. For most papers, I consider it a plus if a well educated 8th-grader would have no trouble understanding the thesis. And unlike Marcus, I am a big fan of arm-chair philosophy and intuition pumping so those who use those methods should hope I am your reviewer:)

As for revise and resubmit, I try to make my suggestions limited so that the author can really focus on what is important. If I have more than 4 major points to make I consider it is too much work for one R&R.

I keep my comments much shorter than Marcus - usually about a page double-spaced. I don't find long comments that helpful because so many of us are given the advice, "If you get a rejection, just send it out again". I think my words have a better chance of being helpful if they are short and give the author something they could quickly change without days of work. If it is not good enough after that change, the next reviewer can offer more advice. I know at least I myself am overwhelmed by long reviewer feedback and am more likely to make changes if I have a few clear suggestions of what to improve.

Lastly, I want to change the culture of reviewing so I make an active effort to be as positive and collaborative as possible. Often I can make a terse point that is not mean but can come across as abrupt. It takes more effort but I read through my comments and re-frame them in a helpful/positive way.

Amanda

In addition to my comment to Stephan - it seems odd that with an over-flooded job market and with far more talent than positions, that so many philosophers that have won in this intensely competitive game make a habit of submitting indecent work.

Helen

I do quite a lot of refereeing. Here's how I approach it:

My first read I just try to get an idea of what the paper is about. I try to resist making a judgment, although in my experience, by page 3 (double-spaced) I get a sense of what my overall judgment is going to be. Still, I want to defer judgment to the end. I am looking for something fresh and new. Refereeing is thankless work that doesn't count on the CV, so my reason for doing it is to help, in a small way, to improve and shape the discipline.

If at the end of my first read I do not find the paper to be sufficiently novel or interesting, I recommend reject. I give a brief explanation (no more than 4-5 lines) for why. If the paper is interesting (good enough to merit revisions), I read it again and make detailed comments. I used to write laundry list referee reports, but currently I try to stick to max 4-5 points (usually 2 major and a couple of minor points). I don't want to micromanage and it is not my job to point out typos (I am a terrible proofreader anyway). My referee report ends up being about 1/2-1 page long.

I rarely find disagreement a reason to reject or require revision, but I do ask for more backup/support if the author makes what I think is an implausible claim.

Referee

I have refereed a lot. I am a mid/late-career philosopher.
1. How do I approach refereeing?: I read the paper through making comments in the margins. I raise questions for myself and the author, as well as make corrections or stylistic changes. (Many of these are not shared with anyone, but form a basis of my evaluation). I ultimately focus on the main points - strengths and weaknesses.
2. As a referee I see my job as offering advice to the editor. I have been asked to referee because I am a published expert in the topic. The editor may not be. My comments need to make sense to the editor, who may not follow the debate, or be aware of recent developments in it. I also want to ensure that everything I write is comprehensible to the author. In fact, I never use the section on comments to the editor only. Anything I write I am prepared to say to the author.
I judge papers on their originality, the strength of the argument, and the effectiveness of its presentation. I hate reading confused papers - I do not like to tell authors that they do not understand the significance of their paper. If the paper is unprofessionally prepared, I am offended that the author would bother to waste my time and the time of other referees and editors.
3. I have recommended acceptance to a number of papers that do not hold my view. That is not an issue for me. The key reason to disagree with a paper is that it is methodologically flawed. The author may think a particular method or mode of reasoning - by analogy, for example - is effective, when the analogy is lousy and misleading. It seems amateurish. Or the author may draw a conclusion with a grand scope when in fact the scope the argument supports is rather narrow. In these cases, provided the paper is otherwise good, I would recommend a major revision - change the scope of the claim, and adjust the arguments accordingly.

Amanda

I am surprised by how many people are "offended" by sloppy papers. If a paper is ill-prepared then I can recognize this rather quickly, and I say that in my comments/reasons for rejection. We are ALL busy people. Some of us sometimes make a mistake and submit a paper too early. I could understand judging someone if he/she does this consistently, but as reviewers we never know if that is the case! It could easily be that the author is going through a really rough time and this is a one-off instance of submitting an ill-prepared paper. My guess is we all will do this at least once in our philosophical career. Given the reviewer's epistemic purview, I think the fair thing is to give the author the benefit of the doubt. (This does not mean overlooking the sloppiness of course. It means kindly informing the author (without malice in your heart) that the paper is not yet publication ready.)

In line with the above, I had a famous professor of mine tell me that he recently received reviewer feedback that said, "This paper could be promising but as of now it is a very messy draft." He went back and looked at the paper and told me, "I realized the reviewer was right, it was a messy draft!" He hence improved it. These things happen. Life is too short, it seems, to get offended about such matters.

Marcus Arvan

I agree, Amanda. I've never been offended by a paper I've been asked to review for being too sloppy. I've reviewed papers that I think are terrible--but i think it's an editor's job to determine what is promising enough to send out to reviewers. As you may know as a reader of the blog, I read a *lot* of intellectual biographies, and you would be surprised at how many great works were considered too sloppy by closed-minded readers who were too easily offended. Frege basically told Wittgenstein the Tractatus was unclear and not worth publishing unless it was broken down into "more exact" journal articles. http://philosopherscocoon.typepad.com/blog/2014/10/some-things-never-change-an-amusing-fregewittgenstein-anecdote.html

Or consider Einstein's early papers or Darwin's Origin. I can give you some truly astonishing quotations from leading scholars during their time who dismissed their work as trash.

Or consider Kant's Groundwork. To this day, I think it's a big mess full of underdeveloped, unclear arguments and bold, unsupported assertions. So what! It's high time we stopped seeing the worst in everything and learn from history that, time and time again, people are way too quick to dismiss things as "sloppy." This is not to say that there isn't offensively sloppy work out there. It *is* to say that we should perhaps be a little bit more humble about dismissing things on those grounds, and perhaps a little less easily offended by these sorts of things. One cannot help but wonder how embarassed Frege or Einstein's and Darwin's most brutal critics might be today were they able to see how wrong they got things--and I think the lesson is that we should all be just a little bit less sure of ourselves and what should offend us.

Referee

Let me try to clarify the offense involved. When students hand in very poorly prepared papers I am offended. I sometimes feel I spend more time on their papers than they do. That should not be the case.
Some papers sent in to journals are quite unprofessionally prepared. That is a shame.

Marcus Arvan

Referee: That's fair I suppose. I guess my concerns here are (A) I've never been asked to review a paper that is as unprofessional as you describe, and (B) I know good work by figures from history and philosophers today whose work "offended" reviewers or readers for being too sloppy--both of which make me wonder whether readers/referees are often far too easily offended and too quick to deem work they don't like substandard or unprofessional.

Amanda

This is an interesting discussion. I think the issue of students is entirely different, but I still am not offended by sloppy student papers. If a student's paper appears poorly prepared, usually that is because of one of three reasons:

1.The student is just not very capable.
2.The student is extremely busy and wrote the paper in a rush.
3.The student simply does not put a high priority on my class or the essay.

None of the above reasons I find offensive. In the first case it is unfortunate college students are not more capable, but that result might have nothing to do with the student and everything to do with various injustices in the school system or the student's personal life. In the second case I try to be understanding. Some students have hectic lives and are trying to work, engage in extracurricular activities, and go to school. I will not grade easily because of this, but I can still extend empathy. The third possibility is not something I am happy about, but I do not believe I am in any place to judge a student's personal priorities. Students might have good reasons to spend more time with family or their job than my class. They will suffer for that with a worse grade, but as long as students accepts this I am not offended by the prioritization. My class is not the be all and end all.

As for it being unfortunate that so many papers are poorly prepared for journals - I agree this is unfortunate if it is true. I am a bit skeptical of whether this problem is widespread, but it certainly happens at times. Marcus brings up good points with some of the best articles in history originally being judged as sloppy. Intelligence is often accompanied by carelessness, for better or for worse (alas, probably for worse). Yet as unfortunate as sloppy papers are, I still am not offended by them. Many misfortunes in the world make me sad, but give me no offense. I reserve offense for cases in which I believe I have been slighted in some way. (i.e. when some injustice ,perhaps minor injustice, has been committed against me).

Referee

Marcus and Amanda,

I think one of the reasons that journals have such high rejection rates is because people now send papers in without much reflection. This is a great shame. The whole system runs slower as a result. Crappy papers are reviewed by multiple journals. And I cringe when I hear that people are getting the advice to just send the paper to another journal. I know this happens because I see lousy papers I rejected for one journal appear more or less the same in a lower tier journal.
I know both of you care about the norms of the discipline. And you want to see change. This should change. People should not submit papers until they are professionally prepared, and the argument is worked out (yes, revisions are possible). And when referees bother to give comments the author should be decent enough to respond to them. I have a colleague who thinks referees' reports are generally worthless. That is not my experience as a published author, nor as a thoughtful referee.

Marcus Arvan

Referee: Fair enough - I think I probably agree. But I also think your points are mostly just additional reasons to transition away from our outdated practices toward the math/physics model where everyone uploads papers to the ArXiv for public vetting before submitting to journals. Under that model, papers are publicly vetted first--which serves to better ensure that people submit papers that are actually good and ready to be published. The problem with our current model is that it is outdated in many ways. Our anonymized review process was created for a pre-internet era when it took much longer to do research. In today's environment, where (A) true anonymity is near impossible to ensure, and (B) scholars are incentivized to overproduce due to horrible job markets and an ever increasing focus on assessment, our old anonymized review model is no longer viable. The problems you raise are just another set of reasons to think so!

Referee

Hi Marcus,

I do think we agree.

There is one issue that I think I either do not understand, or disagree with you about.

"... a pre-internet era when it took much longer to do research."

I do not think that the internet makes it quicker to do research in philosophy. It certainly makes it quicker to get a bunch of stuff together which can be thrown into a paper. But the hard thoughtful work of making a paper out of it takes just as long with or without the internet. That is where corners are being cut. I see it in student papers. And I see it in papers by professional colleagues.

Amanda

Hi Referee,

I think we have some disagreements on facts such as how often really crappy papers are submitted. But persons should definitely make an effort to not submit ill-prepared papers. I just think everyone does it on occasion, and we never know why any individual paper was submitted sloppily so it is best to offer our benefit of the doubt.

The issues of just sending out papers again is tough. If everyone followed that advice, then as you say there are surely papers being submitted multiple times that need serious revisions. I regret some of the work I sent out multiple times as a grad student, and wish someone had told me it needed more preparation. On the other hand, we have all had (probably multiple times) terribly unfair referee reports where the referee was sloppy and did a sloppy job of reviewing. In those cases it does make sense to just send the paper out again. I guess the problem with all general advice is situations are particular.

I agree that research is just as hard, if not harder, with the internet. There is so much information that organizing it all is very difficult. That said I am absolutely in favor of changing the review system in the way Marcus advocates.

Marcus Arvan

Hi Referee: I'm glad it seems we largely agree. However, I'm still not persuaded by you on the matters we disagree about.

You write: "I do not think that the internet makes it quicker to do research in philosophy. It certainly makes it quicker to get a bunch of stuff together which can be thrown into a paper. But the hard thoughtful work of making a paper out of it takes just as long with or without the internet."

I disagree. I remember what it was like doing research before the internet, and it took a *ton* of time to even get the materials (e.g. journal articles) necessary to think through relevant issues carefully. I think that insofar as material is much more readily available--not to mention, it is much easier to get feedback from others via the internet than by ground mail--every part of the research process has been vastly sped up.

I also don't have the clear sense that you do that when it comes to the hard, thoughtful work, "That is where corners are being cut." Were people once much more careful than they are now? Maybe I just haven't been around long enough to witness the relevant changes--but I'm also not sure how we might get clear, unbiased data on any of this. When I read influential articles from decades ego (e.g. Quine's "Two Dogmas", Davidson, etc.), I don't think I see a great deal more clarity, etc., than I do now.

Maybe I'm wrong--but, for my part, I'm much less concerned by perceived "corner cutting" than I am by the persistent use of poor methods (intuitions, reflective equilibrium, etc.) throughout our discipline, both before the internet and today. In my view, if anything needs improvement, that does more than anything. I guess my only other real sense is that--especially in some top-journals--there appears to me to be an increasing emphasis on unnecessary, technically impressive formalisations, and a lessening emphasis on truly bold ideas that push far outside of prevailing paradigms. But these are perhaps my own biases.

In any case, I still suspect all of these things--the things you care about, and the things I care about--would probably be better advanced by the public/non-anonymized math/science review model I've repeatedly advocated.

Amanda

Okay so we are all wondering what the weird thing going on is with the Helen/Marcus name thing....

MARCUS ARVAN

Oops! I re-posted one of Helen's comments yesterday that got caught in Typepad's spam folder, and somehow my browser remembered her name rather than mine. Fixed now - thanks for letting me know!

Amanda

haha I knew there had to be some explanation like that. I briefly thought maybe there was some elaborate cover-up and you and Helen had been the same person all along...but then I realized Helen is definitely real and that didn't make any sense!

Stephen Bloch-Schulman

Folks,
Sorry to be late to respond.

OH posted: "I'm curious- could you elaborate on how a paper's poor quality is reflected in its bibliography?"

The problem appears when it is missing work by the interlocutors that might speak to a specific thesis. I don't expect an author to know every reference out there, but there is a sense that one should know other arguments and other work that is often cited and speaks directly to the argument at hand. And I find that this lacuna is more common than I would have expected before I did so much reviewing.

And, I agree with Amanda above that many folks are not good at writing abstracts. I hope I made it clear: when I read, I really want to find good work. And it is often evident quite quickly that what is submitted won't fit the bill. But--and maybe I didn't say this enough, and if not, I appreciate being pushed to be clear here--just because it doesn't look promising in the beginning doesn't mean there isn't something there, in the end, and I, of course, read the whole and read it carefully and read it always hoping that there is something, even if not the whole, then some nugget of an idea that can be expanded on or has real promise. So, when I said that I can often tell early, it is not to say that I don't still hope for better and read looking for better. It is to say that I am surprised by the number of papers that quickly seem untenable and then, after a full and complete reading (and often re-reading and re-re-reading) don't offer anything else.

Hope that clarifies.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)

Open job-market threads

Categories