Our books






Become a Fan

« Do the little things matter on the market? | Main | Propriety, good structures, and justice -- thoughts on leadership in academia »

07/02/2019

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Sam Duncan

For what it's worth the Journal of the APA asks authors to give a list of people who shouldn't be solicited as referees for reasons like that. Of course that depends on authors being honest (and unfortunately they often have incentives not to be). But it's better than nothing and it would be a very easy thing for other journals to do. Also, while I'm on the giving credit where it is due kick I think Analysis and Ergo also have triple blind review policies.

Humanati

I'm completely in favour of taking further steps to ensure anonymized review. Personally, I don't even give the editor *the option* of having me review a paper if I know the identity of the author. I just tell them that I can't do it, and suggest alternative referees. A deeper worry that I've long had with the review process--one that I wish were discussed more often and openly--concerns the role of editors and associate editors. I'm not sure how representative this is of the situation at large, but I've heard about a disturbing number of cases (many first-hand) of (i) (assoc) editors intentionally assigning particular referees to authors whom they know/like in order to increase chances of a favourable report, (ii) supervisors in editorial positions taking certain (morally dodgy) steps to help their students get papers accepted in high profile venues, (iii) editors deciding to accept a paper that received lukewarm reviews primarily because of the author's reputation/prestige. This might be a slightly cynical response to the situation, but these stories sometimes lead me to be far less impressed when those who are particularly famous (or even just very well-networked) manage to secure publications in high profile venues.

Amanda

Anonymous review in philosophy is not a complete joke, but it is getting close to a complete joke. I say not complete because it is still anonymous maybe 50% of the time, and if someone is a no name it is more likely to be anonymous. So it does give some path to help the disadvantaged. However, it only gives that path alongside a huge extra advantage to the already advantaged, who routinely have their paper reviewed by people they know. Some other thoughts:

1. The anonymous review statements in journals are just silly. They are not taken seriously, and people routinely admit to violating blind review in public, with no fear of professional repercussions.

2. I have had many conversations with people in the discipline who admit to violating blind review. There have been many facebook posts where people publicly admit to violating blind review. I have heard from more elite professors that they *purposely* send their papers around to their friends because they *want* to violate the blind review process. Those same papers are routinely published in top 5 journals.

3. In spite of the open secret that blind review is violated all the time, the profession then speaks out of both sides of its mouth and claims that publications are this one pure measure of merit, not influenced by prestige. That is ridiculous. Prestigious people, from prestigious places, in prestigious research circles, have a *huge* advantage in publishing in the top journals, an advantage that is completely divorced from merit and based only on the fact that they get papers reviewed by their friends, and others do not. (This is not to say that prestigious people are not more talented on average, maybe they are, but regardless they still have this advantage.)

4.Of course it is a problem! If it is really impossible to have a blind review process these days (not sure about that, but suppose it is) then we should not put on the facade of blind review. We should have some other publishing system, and be honest about it.

5.The system we have now literally allows people to purposely send their papers to their friends and have them published in the top journals under the guise of blind review. (of course this doesn't happen every time, but it happens frequently.)

6. I had a senior figure tell me that he recently refused to review a paper because he was too busy. He laughed about how he had used the excuse of knowing the author, while simultaneously admitting that this is not the real reason, as he reviews papers of people he knows all the time.

7. I am not sure how well the steps to try and keep things anonymous would work, but I am all in favor of trying - at least until we have a different system. I do tend to think we need an entirely new system. However, more efforts to keep things blind will probably help a little. My only worry would be that it might just make people be more dishonest about what they do. What I like about social media is it has finally made it common knowledge that the blind review system is routinely violated. It used to bother me when maybe 5 years ago I would talk about this and the naive people wouldn't believe me. Today it is very hard for any minimally informed person to believe blind review is mostly blind.

Reader

Marcus,
Who is writing the story you link to? I could not figure it out. Are they even a philosopher?

Marcus Arvan

Reader: The author is Simine Vazire, Professor of Psychology at UC Davis. https://en.wikipedia.org/wiki/Simine_Vazire
https://profile.typepad.com/6p019b0070529b970b

She was editor of Social Psychology and Personality Science: https://en.wikipedia.org/wiki/Social_Psychological_and_Personality_Science

I am not sure why it is relevant whether they are a philosopher. Peer-review is peer-review, irrespective of discipline.

Marcus Arvan

Amanda: in my view the most disturbing thing about the online discussions you are referring to (which I have seen too) is just how many people seem to openly rationalize these actions. Look, if you think violations of anonymized review can be justified, then fair enough (though I disagree). But in that case let’s be clear and honest about what’s going on: that we don’t in fact have a system of anonymized review. Further, in that case, as the former editor at the piece I linked to notes, then it seems like we should be much more skeptical of the idea that publishing is determined by merit—as a system where people know the identities of authors a significant proportion of the time is open to non-merit-based biases. Basically, I just want moral consistency. Either do more to ensure anonymized review, or do away with the fiction that peer-review is anonymized.

grad

I don't see how this can be solved without also mandating that no paper be submitted to a journal after it has been presented at a colloquium or conference. I can think of at least one paper that I can't imagine getting through peer review if not for everyone knowing who wrote it because it had been delivered as a talk a dozen times in the previous few years. And that version of the problem is mostly for people at the very top, people who get invited to keynote conferences and give departmental talks. You cant stop them from pushing their crap papers through by making sure everyone (not just their friends they circulate it to) knows it's by someone famous!

tenured but shy

In the past I've disclosed that I know who the author is, and let the editor decide whether to go ahead anyway. But I am persuaded that I shouldn't do that in the future.

Agree with Humanati that making submissions anonymous for editors is very important, especially if they are making desk-rejections but also otherwise.

I am honestly flabbergasted by some of the stories Amanda is sharing. I guess it helps to explain why some bad or just plain dull papers end up getting published in "top journals".

Marcus Arvan

grad: that’s one of the reasons I advocate abandoning the anonymized peer-review model in favor of an open ‘ArXiv’ model. In addition to presenting papers at lots of colloquia and conferences, I’ve noticed that some well-placed people tend to openly post titles and abstracts of their papers under review on their websites—something that makes it easy for a curious reviewer to figure out they wrote it through a simple Google search. This is yet another way that anonymized review may be routinely compromised.

Still, I think your main claim—that a particular paper wouldn’t have made it through anonymized review given that it was presented over a dozen times—is almost certainly false. Sure, there may have been quite a few specialists present at those presentations—but surely, of the thousands of philosophy PhD’s out there, there are still plenty of people who never saw the paper. For my part, I am routinely surprised by the number of *published* papers (even ones in very good journals) that specialists I have met in the relevant areas have never heard of. There will nearly always be referees in a given area who don’t know the author of a given piece—so as long as our journals and discipline more broadly say they/we practice anonymized review, it seems to me that more care should be taken to ensure that people who haven’t seen the paper before aren’t reviewing it. For, again, allowing people who know a paper’s author to conduct a review completely defeats the point of anonymized review, which is to prevent knowledge of an author’s identity from playing a role in editorial decisions.

A Philosopher

Re: Marcus and grad:

"Sure, there may have been quite a few specialists present at those presentations—but surely, of the thousands of philosophy PhD’s out there, there are still plenty of people who never saw the paper."

Marcus is right, but grad's point raises another issue I don't often see mentioned in these discussions. The referee pool for papers is typically very small: e.g., just the "famous" faculty and their rising-star graduate students from top departments. It's plausible that usually many, or perhaps most, of *these* people will have heard (of) the paper by the time it's under review.

This issue highlights the inbreeding of the system: those working at top research schools edit the journals, they select their friends (who are also at the top research schools) to do the refereeing, and those referees routinely just so happen to like the quality of the submissions from their own friends at other top research schools.

In any case, I finished my PhD several years ago. In all this time two journals have asked me to referee a paper (each just one paper). So I've refereed two papers total. But I see plenty of f'ing *grad students* from top-20 schools with good connections who list refereeing for like a dozen journals on their CVs. It's not hard to guess what sociological mechanisms explain this. It's certainly not the case that someone who just started studying specialty X a year or two ago, no matter how talented, is better positioned to referee a paper on X than I am, given that I've been reading the literature for 3-5x as long as them.

I am constantly discovering not only *papers* in my AOS which I had not seen before, but *philosophers* who work on the topic. There are thousands of professional philosophers in North America alone. For any given AOS there will be at least hundreds of people working on it. I just don't believe that when an editor says they can't find someone who knows X to referee a paper they've really checked the *hundreds* of potential referees with an AOS on X. (If grad students are qualified to referee papers, something I myself mostly doubt, than the pool is even larger.)

The point is that (1) refereeing is largely done by an editor's friends or other well-known people who easily come to their mind, (2) these people tend to be from research schools, or at least have a strong network in those places, and (3) it's no surprise that the "top"-heavy referee pool largely knows the submissions of their friends, even with their names removed.

So an obvious way to at least partly mitigate the problem is for editors to expand the referee pool beyond their social network.

Marcus Arvan

A Philosopher: that is a very nice observation. I’ve heard anecdotally that editors may select people from top programs as reviewers on the rationale that they are the best philosophers and therefore likely to be the best reviewers (qua having the highest standards). However, the issue in the OP raises a problem for the inference here (the “therefore”). For even if we suppose that people from top programs are the best philosophers (which I don’t mean to suppose, but let’s assume it for the sake of argument), if those reviewers are reviewing the papers of people they *know*, then there’s a serious conflict of interest. Even if they are the best philosophers, we should not want a peer review system that permits people to knowingly review their friends. The answer, I think, is that journals shouldn’t assume that a small group of people from top programs are likely to be the best reviewers with the highest standards. Rather, journals should think a broader reviewer pool will be less likely to compromise anonymized review, and therefore likely to be less biased than a small group of people who mostly know each other’s work.

A Philosopher

Marcus, that's a good point. What qualifiers someone as a good (or the best) referee needs to consider not only philosophical skill, but also things like whether their review really will be blind. (Of course, as you and others point out, plenty of editors seem to disagree on this point.) I think it's worth pressing on the idea that the best philosophers are at the top places. Even if this is a real general trend, it still leaves room for dozens of highly skilled people, "down below", on any given topic. I think we can all think of many very good philosophers (old friends from graduate school, etc) who now teach at community colleges or regional public schools or whatever. I know I can. (Yes, of course, I can also think of many really, really lousy philosophers as well.)

But in any case, more goes into good refereeing than just raw philosophical skill or talent. You need to know the literature and yourself have reasonable standards for review (e.g., not be one who's apt to "axe grind" or get hung up on your own pet ideas). On the former point, I think there's a clear case to be made that graduate students are, for the most part, not well positioned to referee. They're still learning the literature. On the second point, I have no reason to think that "top" people are more apt to axe grind, aside from worrying that those in privileged positions who rarely have their privilege questioned don't normally reflect much on their own idiosyncrasies. But in any case, if we can't get away from axe grinding, at least we should spread the bias around by letting people from a variety of perspectives grind their axes. A referee's job isn't (or shouldn't be) to weed out all the wrong ideas; it's to weed out ideas not worth discussing. What's worth discussing is a project the broader community should curate, not just a few people at the top.

I know I've gotten off topic, but I do think there's a strong case to made for spreading the referee workload around. In any case, if you're an editor who won't ask someone at a community college or regional public school to referee, you shouldn't complain about a lack of referees. It's a problem of your own making.

Amanda

Yes I agree Marcus, what bothers me the most is not any one thing in particular, but the combination of things:

1. Journals claim, with an air of serious authority, that blind review is a strict protocol.
2. In spite of 1, editors not only fail to go to efforts to find a reviewer who likely is unaware of the paper, but they give permission for someone to review the paper when that someone *admits* they know the author.
3. Blind review is touted as an honest and fair measure to judge real talent, under the guise that it is quality and not connections that get papers published.
4. People not only publicly admit to violating blind review, but then argue against efforts to make reviewing more blind, insisting, with a straight face, that they will be "fair." (In spite of all the evidence that shows people with the best of intentions cannot be fair.)
5. If someone who is not famous complains about peer review, they are accused of having sour grapes.

Anyway the point is that either we should make serious efforts to keep blind review blind, or we simply should not claim it is blind. We should just call it peer review, and leave the blind out of it. I would prefer this. Yes, there would be biases in review, but the biases already exist. And in this system about the same number of papers would still be blind review, because by chance those less connected in the profession will still get people who don't know them to review their paper.

The suggestions about widening the review pool make sense, but I have zero hope it will ever happen. This is the response you will get:

1. It is already really, really, hard to find reviewers. (when you point out that it won't be hard if you widen the reviewer pool, this will be ignored.)
2.Editors are already overworked in an underappreciated position.
3. Papers already take too long to go under review.
4. We can't trust that some random person who lists such and such as a specialty on their CV is actually an expert.

I really think the best hope is doing away with blind review. Short of that, things like listing who has read the paper might help. But it might just backfire. The more famous one is, the more likely one is to just post the paper on their webpage or facebook and then far too many people have read it to list them. But at the least editors can refuse to let someone review the paper if they admit to knowing the author. (but this will get the complaint about review times and finding referees.)

A Philosopher

"4. We can't trust that some random person who lists such and such as a specialty on their CV is actually an expert."

I have my papers reviewed by non-experts all the time. The last referee report I got a few weeks ago had a line like "The author makes empirical claim P about X; I don't know the empirical work on related phenomena Y, but P seems to imply Q [where Q = some true empirical claim] about Y, which I imagine is false". This wasn't obscure stuff either, but mainstream empirical work.

So, yeah. Maybe this paper was reviewed by some grad student at NYU who just started working on the topic last semester. Who knows.

In any case, to those editors I should say: I'm not advocating that you ask people who haven't published in 20 yrs to review papers. But if someone got their PhD within the last few years from a ranked program and because of bad luck ended up at a community college, I think we can all safely assume they're competent to review papers... They should get some sliver of professional respect as a researcher from the community. But maybe I'm asking too much when there's some wiz kid at Rutgers who doesn't know what he doesn't know who we can get to do the work instead.

Amanda

A philosopher: fyi I agree with you, I was just listing the reasons that people will give. I think they will give reason 4, even though it is a bad reason :(

Maybe you knew that, but thought I'd clarify.

A Philosopher

Oh I know, Amanda. Just venting. Thanks though for the clarification. (And I fully agree with you too.)

grad

Marcus: I'm not sure I understand why the considerations you raise tell against my claim. My assumption is that the reviewers assigned the paper in fact were among the rather large but (as you rightly point out) not universal set of people who knew who the author was, not that it would have been impossible to find someone who didn't. That the reviewers knew who wrote it but were happy to review it anyway without telling the editor, or that the editor knew that the reviewers knew who wrote it but was happy to let them review it anyway, is speculation on my part, but I'm reasonably confident that it's right. The fact that it would have been possible for it to have been reviewed by people who didn't know who wrote it does not entail that it "almost certainly" happened that way and not the way I suggested!

As to the broader point, I just think that so long as reviewers are on the honor system vis-a-vis refusing to review papers when they know who the author is the problem is much larger than avoiding people the paper has been circulated to; it sometimes also involves the much larger group of people that will have seen it presented. If we stipulate that reviewers will stop agreeing to review (and then accepting) papers when they know who the author is, yes, genuinely anonymous peer review is possible. So long as things remain the same, just asking the author to disclose the people to whom the paper has been circulated won't be enough if it has already been presented multiple times, especially if there is buzz about what so-and-so is working on now.

Nicolas

A Philosopher: I don't know, I've refereed at least half a dozen papers every year for 5-6 years now, and I don't fit your description. I've also heard repeatedly that editors rely heavily on not just grad students but junior faculty, postdocs, etc., I take it, most of whom not from top programs; that they find referees by searching e.g. Philpapers. Given what we know about how hard it is for them to find referees to agree to review submissions, I very much doubt that they rely on the sort of referee pool you suggest.

More broadly, this thread is taking a conspiracy turn that I find a tad worrying. We don't have any data—everyone is just sharing hearsay and anecdotes. But I'd bet that the overwhelming majority of peer review is done anonymously. I find that most (though surely not all) papers published in top journals are actually pretty impressive. And we've all heard, including HERE, many stars share they personal horror stories of peer review (which could not happen if the story brushed here was true).

None of this is to deny that, yes, sometimes truly anonymous review is compromised, but let's not overstate the problem.

JR

I always say in the cover letter who have commented on the paper and who should not review the manuscript for this or other reasons. I have always thought that every one does this!

After receiving a desk-rejection from a top journal with the editor's justification for it: "it is impossible to find highly qualified reviewers in a timely fashion for all the manuscript we receive, so we have dediced to reject this paper" I have also started to suggest the names of qualified referees who I (honestly) think would be good referees and who have not seen the manuscript.

When reviewing papers, I have declined because I've know the author. Sometimes I find out who the author is while reviewing the paper (journals don't always properly blind papers) then I have completed the review and said to the editors that I know who the author is but completed the review anyway and that you should get another opinion as well.

Marcus Arvan

Nicolas: I appreciate your concerns about this discussion being based on anecdotes. However, someone (I can't quite recall who) did an empirical study on refereeing in philosophy not too long and found (if I recall correctly) that journals tend to use a relatively small pool of referees who complete the vast majority of reviews. One takeaway of the study (again, if I recall) was that there is a vast pool of qualified reviewers who aren't being used that often. If anyone reading this can recall who did that study and link to it, I'd be appreciative! In any case, even in your case you mention reviewing maybe 6 papers a year. However, I know people on social media who have said publicly that they review *dozens* of papers a year. Which is basically what the aforementioned study found: that even if people like you review 6 papers a year, a much smaller pool of people is reviewing far more than that.

Anyway, anecdotes aside, I'm not sure this thread is conspiracy-like. I have no doubt that a majority of papers are anonymously reviewed. But I think there are reasons to worry that a fair proportion are not. Amanda works at an R1 and has said she has heard people say that they engage in sketchy practices--and the OP I wrote is based on several facebook threads I've seen in just the past year or two where a significant proportion of commenters *openly* said they review papers of people they know or leave it open to editors to decide (and in those cases some editors even chimed in saying they sometimes permit it).

In any case, I agree with you that we shouldn't overstate the problem. I just don't think we should ignore it, given the evidence we do have.

Nicolas

Marcus, 6 is averaging over the last 5-6 years. It's been increasing. Recently it's been closer to a dozen, not counting those I turn down. I'm not saying this is a lot, simply that reviewing a fair number of articles, many more than A Philosopher says they have, doesn't just happen to elite grad students and/or professors.

Regarding your other points, fine if that can be substantiated. Until then, beware the availability heuristic and confirmation bias.

Marcus Arvan

Nicolas: that's fair!

A Philosopher

Nicolas, it sounds to me like you're not interpreting my comments very charitably. For example, by "elite" or "famous" I obviously don't just mean people at top-5 programs or something like that. But we all roughly have some idea of who the "cool" kids are in our areas, and although they tend to be faculty at top-5 or top-10 or whatever programs, they can be from all over. Do you teach at a community college? Do you teach at a small regional public university or liberal arts college where 90+% of the students come from within a few hours drive and most of those admitted are in remedial English? Are you a recent PhD graduate who's either unemployed or adjuncting? If not, then you're not a counterexample to my claim. And even if you were a counterexample to my claim, well, of course I wasn't claiming that you can't find any counterexamples. Also, the fact that editors sometimes go through philpapers to find referees doesn't mean much, as they still could be mentally filtering their results through their various prestige biases.

Sorry, but when we're discussing issues which materially affect my livilihood, I don't have much patience for people who don't seem to have tried hard to find charitable ways to interpret the personal stories of those "losing" out in the profession. I could be very wrong, but the fact that you get over a dozen requests to referee a year suggests to me that you aren't unemployed or adjuncting.

Nicolas

A Philosopher: Not sure who’s not being charitable here, but I understand your frustration and honestly don’t have much of a good explanation. I didn’t get my PhD in the US, and where I did does not by any stretch count as a top program around here. Most people haven’t heard of my advisor on this side of the ocean. I was on the market for four years. Sure I did have luck and success, but the reason I’ve been asked to serve as referee, I suspect, mostly has to do with my qualifications, not prestige or connections or whatever. I may be wrong, but if I am, trust me, those haven’t served me particularly visibly while I was on the market! Nor have they guaranteed me any publication. Just got a new rejection today! If I’m privileged we’ll damned I’m not lucky, seems like I’m choosing the wrong journals.

Again, I have a hard time believing editors would not appeal to a large pool of referees based on qualifications rather than prestige given how hard it is for them to find referees.

You can check my vita here if you’d like, nothing to hide: nicolasdelon.com

A Philosopher

Hi Nicolas, thanks for the reply. Don't take this the wrong way, but you are quite clearly one of the "cool" kids whom journal editors are seeking out. I'm not surprised at all that you are now getting a dozen+ requests to referee papers. You are an impressively accomplished philosopher for only being a few years out, and I mean that sincerely.

But there are *a lot* of philosophers out there (literally hundreds) who have CVs that look nothing like yours. Their CV doesn't include names like "NYU" and "Oxford", anywhere on it, in any capacity, and they aren't being invited to contribute to publications and aren't asked to edit philpapers sections. But many (of course not all) of them are competent researchers and want to be members of the philosophical research community and taken seriously. They end up working at community colleges and in other similar gigs, because (well) there simply aren't enough jobs out there for everyone. I would link to some CVs or faculty bio pages or something to give you an idea of the sort of profile I have in mind, but of course I won't put anyone under that sort of spotlight.

The point is that many of these people are perfectly capable of refereeing a journal submission. But they don't get asked to, or they are very, very rarely asked to.

I don't see what separates those asked to referee from those invisible to journal editors other than prestige markers. For example, the bar for refereeing a submission can't be that you're a faculty member at a 4-yr school or have x-number of publications, since evidently grad students are often asked to referee. I mean, honestly. I think anyone who takes a serious look at the people asked to referee, vs those not asked, will see the only trend is that various prestige markers exist in the one but not the other camp. Of course, these markers vary *a lot*. Perhaps you were never asked to referee as a grad student because, as you say, you didn't come from an "elite" North American school. But now you have other prestige markers on your CV (again, no need to list them out), so you get requests.

Actually, I'm not sure why your own case doesn't support my point. Were you not qualified to referee papers back when you first finished your PhD? You yourself said that you were rarely asked to referee before the last year or two, and it's only been in the last year or two that you've racked up many prestige markers and have been around long enough for your name to percolate through the philosophical ether. So your case fits pretty well with my hypothesis that the referee pool is mostly a function of prestige markers plus social visibility of some sort.

So here's a way to put the point: there are various prestige markers (e.g., grad student at top-10 school, "famous" person who gets invited to speak at the APA, publication in Nous, etc etc) and who gets asked to referee seems to track pretty well with these markers. But there are competent philosophers out there who lack these markers (e.g., late graduate students and recent grads from low-ranked programs who haven't secured permanent employment or a fancy post-doc, but others as well) who could referee, but are never asked.

All I'm saying is that if editors wanted to relieve their referee shortage and wanted to help mitigate the bias problem of referees recommending their friends' "anonymous" papers, they could reach a bit and start asking some of these philosophers to referee.

And, by the way, this is all orthogonal to questions of merit. Let's stipulate (as is surely not the case) that the various prestige markers really do correlate closely and cleanly with philosophical ability. Everyone acknowledges that there are more talented and capable philosophers out there than there are available jobs, journal slots, etc. So there are talented, capable philosophers who could referee but lack prestige markers due to scarcity. The margins in talent and competence which separate those who make it from those who don't are, we all should acknowledge, razor thin and often a function of context. Surely these same margins don't separate who's qualified to referee.

Amanda

Nicolas: you accuse other people of relying on heresy anecdotes, and that we sound like conspiracy theorists (btw public facebook posts that can be easily accessed are not heresy. When I hear direct testimony, that is not heresy for me, either...it is direct testimony.)However, then you state, *with no evidence at all* that you "bet" the vast majority of papers are blind reviewed. At least testimony is some form of evidence. Relying on what you "bet on" true is not any evidence at all.

More, to counter what a philosopher has said, you give your own anecdote about the papers you have reviewed. For some reason you trust anecdote when it comes to your own experience, and criticize them when it is about the experience of others. You do the same thing with mentioning papers of famous people that have been rejected. (And of course those stories are consistent with our claims. Just because famous people often have advantages in peer review doesn't mean all of them do, much less for every paper they submit)

Also, I am curious why someone *would not* believe the claims Marcus and I have made about facebook. Facebook is a very public forum, with many philosophers having over 1000 friends. Marcus and I are not facebook friends, so the sources are independent. Do you think we are really just making this up? If Marcus is that much of a liar not sure why you would read this blog. Or do you think the fact that many people admit to repeatedly violating blind review, in public, is not any credible evidence that there is a problem? Why wouldn't it be evidence? Do you think the people who make these claims are lying? If we took blind review seriously, wouldn't people be afraid to make these claims? Do you think the people who make these claims are an exception to the rule? That would be odd, because when a claim is socially unacceptable in any way (like admitting to violating blind review, although not as socially unacceptable as |I would like) then the general tendency is that far *less* people admit these wrongs. Not that far more people admit them.

I just don't think your interpretation of of the information is sound, epistemically, even putting aside the ad hominem

Alas, I am afraid that people who are convinced the system works are not going to change their mind. I am more interested in people who are agnostic.

This reply was a bit harsh. But I think (1)the repeated violation of blind review is a problem that should not taken seriously, and at the very least we should not deny it happens. I say this from a privileged professional position. Also, (2) it drives me crazy when people accuse others of having poor evidential standards because they rely on anecdotes, and in the same breath give evidence for the opposite claim using their own anecdotes. It is even worse than the specific accusation is that the other side is sounding like "conspiracy theorists." Really? Direct testimony from multiple independent sources about this happening is a conspiracy? IF you don't believe me, then go ahead and call me a liar, but there seems no plausible way of calling multiple sources of testimony a conspiracy.

Amanda

opps, voice to text. I of course meant "should be taken seriously"

Marcus Arvan

Amanda, Nicolas, and others: I’d like to ask everyone to bear this blog’s mission in mind, and dial down the rhetoric. I’m happy to host spirited debates on the blog, but some recent comments are in my judgment veering beyond the blog’s mission. From here on out, I’m only going to post comments that have more measured / less personalized language.

A Non-Mouse

A Philosopher writes: "I don't see what separates those asked to referee from those invisible to journal editors other than prestige markers."

What you don't see might be something that tends to occur with having prestige but that tends not to occur without having it, such as a "special understanding" of a topic (where such an understanding can be had by only those with significant talent who have carefully and thoroughly studied the topic). We should all agree that having a PhD on topic X does not guarantee a special understanding of X, because there are many weak or insignificant dissertations. (Here, I'm assuming it's very likely that one would not write such a dissertation if one has a special understanding of the relevant topic.) Of course, prestige doesn't guarantee such an understanding, but the likelihood of having it increases with prestige. We should all agree that we want referees to be as credible as possible, and that those with a special understanding are the most credible. If so, we should want referees with prestige, though not because they have prestige. Instead, it is because they are most likely to have a special understanding. For this reason, even if one thing that distinguishes those asked to referee from those invisible to editors is prestige markers, this by itself should not bother us.

Even if in addition--as you write--"[t]he margins in talent and competence which separate those who make it [into prestigious positions] from those who don't are...razor thin and often of function of context," why should we be worried or upset? The only reason of which I can think is that the thin margins mean that the likelihood of having a special understanding is not much greater for those with prestige than for those without prestige. But it seems that the likelihood is much greater. This is because when you consider the differences in workload between those with prestige and those without it, it becomes clear that those without it likely won't have as much time and energy to maintain their special understanding as those with prestige (if they had it at all). So again, I'm not sure there is any reason to be bothered by all of this. All fo this lends support to the idea that prestigious referees are the way to go.

I not comfortable with the conclusion I've reached. But the argument has brought me to it. If I've made a mistake, I'd like to know what it is.

Nicolas

Amanda: maybe I’ll reply to your post substantively tomorrow but I don’t understand why you’re reacting like that when I haven’t accused anyone in particular of anything. I started by saying ‘I don’t know’, shared my experience without making sweeping claims, acknowledged that all we have is hearsay (not ‘heresy’ (sic)) and anecdotes—I said WE, not YOU, so yes, that’s including me. Facebook can be full of crap. Sure, reliable testimony too, but not something we can infer much from. Again, I’m not asking anyone to infer anything from my case. I simply registered my surprise given my experience and well, what I’ve heard, read and what I think is a plausible picture of the system. You don’t have to agree, but you have to let me speak.

As for conspiracy, I did find that the sound of the thread was veering toward something more conspiracy-like than I think is the case. I just think the process is more random, haphazard, unstructured and poorly organized than some of the comments let it sound. Anyway, I wish you’d made your point more respectfully, as I did. It’s not the first but I hope it’s the last time.

All that being said, I agree the system is far from perfect. Whether we’re well served by claiming it is total bullshit seems like quite a different question to me.

A Philosopher

I apologize to Nicolas and Marcus for pushing the line. I think the issues we are discussing involve an inherently messy personal and social component which is difficult to disengage from the purely professional issues, but I respect the need to keep the discourse professional and (of course) the need to follow Marcus' policies.

In reply to A Non-Mouse, my reply is what I said before: "So there are talented, capable philosophers who could referee but lack prestige markers due to scarcity." There are, in fact, people with "special understanding" who find themselves unable to snag a prestigious job, publication, invited APA talk, etc, simply because there are more people with special understanding than these opportunities. But perhaps more importantly, one of the upshots from the above discussion was that these people with prestigious CVs may not be the best referees, since their close connections often lead to refereeing that's not blind. A peer review system which includes not only the *most talented* referees, but everyone who's competent to do the job, may mitigate this problem and also include a diversity of viewpoints which strengthens the overall product.

A Non-Mouse

A Philosopher: Thanks for the response. If I understand your response, it is this: there is a reason to worry about the high proportion of prestigious referees because the circle of prestige is so small that having a high proportion of prestigious referees compromises blind review. You then want to suggest: given the worry and the fact that there are some referees of the highest quality who are not prestigious, editors have most reason to select non-prestigious referees.

This is a reasonable response, but I think it is misguided. The main purpose of blind review is to ensure high-quality publications. So, if editors have most reason to select non-prestigious referees because of the compromising of blind review, they have these reasons on the basis of promoting the aim of ensuring high-quality pubs. But selecting non-prestigious referees will not promote the aim. This is because they cannot be confident that their selections will do a good enough job to ensure high-quality pubs. Let me explain. They can be confident that there is a tiny proportion of non-prestigious people who have the relevant special understanding (given what I wrote in my previous comment), and so editors can be confident that they will likely not select one of them when choosing randomly from among those with the relevant AOS. They have to have prior knowledge about the pool of non-prestigious referees in order to be confident in their selections. I doubt that they usually have this sort of prior knowledge. So, it seems, they can be confident that they will select a merely competent referee from the non-prestigious pool. But merely competent people will likely not have the talent to provide the sort of feedback that will ensure high-quality pubs. Thus, editors can be confident that when selecting from those without prestige, they will select someone who will not have the talent to provide the sort of feedback that will ensure high-quality pubs. For this reason, editors have very weak (if any) reasons on the basis of ensuring blind review, to select non-prestigious referees.

If this is correct, then editors have most reason to select non-prestigious referees only if the quality of pubs is compromised more by the small circle of prestige than by selecting non-prestigious referees. For good reason, I think, editors will not believe that the quality is compromised more by the small circle of prestige than by selecting non-prestigious referees. Since the disagreement between you and I (I think) rests on whether editors have good reason not to believe it, I'll offer an argument toward the conclusion that they should not believe it:

P1. Pubs of the highest quality, those that can significantly advance debates, are already in short supply (not because of the compromising of blind review, but because such work is too difficult for a great proportion of us to do).

P2. If so, then it is very important that the work that does get published is as high a quality as it can be, but referees who are merely competent are likely not to have the talent required to offer feedback that will improve the quality of the work.

P3. If it is very important that the work that does get published is as high a quality as it can be, but referees who are merely competent are likely not to have the talent required to offer feedback that will improve the quality of the work, then the quality of pubs is seriously compromised by selecting non-prestigious referees.

P4. If the quality of pubs is seriously compromised by selecting non-prestigious referees, but is not seriously compromised by the small circle of prestige, then the quality of pubs is not compromised more by the small circle of prestige than by selecting non-prestigious referees.

P5. The quality of pubs is not seriously compromised by the small circle of prestige (because prestige tracks the talent required to ensure high quality pubs).

Conclusion: the quality of pubs is not compromised more by the small circle of prestige than by selecting non-prestigious referees.

The most controversial premises, I guess, will be P3 and P5. What do you think? What argument can be brought against any of the premises?

A Philosopher

Hi A Non-Mouse. I don't really have the time or energy to carefully go through everything you say, and I'm not sure it's worth the effort (for either of us) or that this is the place to do it. I disagree (to varying degrees) with most of the premises and presuppositions involved in your response (and not just the numbered ones). I definitely do disagree with P3 and P5.

Regarding P5, I'm not sure what to say besides what's already been said: the small circle of prestige leads to too much unblinded refereeing, whereby a lot of work gets through on the reputation of the scholar alone. Philosophers, no matter how talented, are people with biases. (The only way around this is either to ensure blind refereeing, or go with unblinded *public* refereeing whereby there's enough people for the biases to wash out a bit in the marketplace of ideas.)

Regarding P3, I don't think the function of journals is to publish only the very best. They are a forum for holding professional philosophical discussion and scholarship. They should host all philosophical discussion and scholarship of professional quality that's worth discussing, not merely the very best or something like that. I'm actually not sure there's any other discipline that views their journals as a home only for "the very best". Every discipline has a few "prestige" journals, but by-and-large journals are the place where the daily work gets done and spread. The ultra-low acceptance rates in philosophy journals (and the aversion to taking seriously anything published in a journal with higher rates) aren't typical in most science journals, for example. Science journals generally publish any competent work that contributes to the literature.

More generally (following up on those thoughts), I think your focus on talent and "special understanding", along with the focus on journals only publishing the highest quality pieces, shows a fundamental disagreement between us on the aim of publication. At the risk of reading too much into what you're saying, it sounds like you think the role of referees is to guard against wrong ideas or bad arguments getting into the literature while helping improve and perfect the correct ideas and good arguments, with the end result being that what's published is only the philosophical truths (as best as our best people can figure them out). Instead, I think journal referees should merely serve as a check of some basics, like standards of scholarship (has the author cited the relevant literature? does their work demonstrate expert-level competence of the material? is their idea original? are they misconstruing the views of others? etc) and minimal standards of philosophical competence (do they avoid obvious fallacies? are they equivocating in a way that any expert would catch? etc). I also think referees, as experts in the field, should make some judgment about the overall interest of the paper: is this an interesting or original take on the question at issue? Is it an interesting question? Does this make some contribution? etc. Thus, the role of referees (on my view) isn't to shepherd into publication correct philosophical theories while keeping out every wrong idea, but instead to curate a selection of professionally competent, philosophically interesting articles. I certainly don't have room to defend this normative claim about how journals should be run and how referees should do their jobs (obviously many people disagree with me), aside from making a simple point. Philosophy is way to hard for the two or three people involved in refereeing to get it right in the one pass that is the normal submission evaluation. So attempts to "shepherd truth" into the literature will generally fail. In practice, what actually ends up happening is that really interesting philosophical discussions that should see the light of day and get carried out in public instead happen in private between a submitting author and the 2-3 referees critiquing them in rejection reports. (And for whatever it's worth, I think even the more "minimal" standards I'm advocating would still rule out 60-70+% of journal submissions, as many submissions *aren't* professionally competent and don't meet minimally scholarly standards.)

To tie this back to the blog topic: I think we can mitigate some of the problems of unblinded refereeing by widening the circle of referees beyond prestige people, and such a loosening of who we ask to referee is perfectly in line with the sort of view of refereeing I defend in this response.

A Non-Mouse

A Philosopher: I don't understand your objection to P5. Still, in general, your latest comment is very helpful in helping me to understand our disagreement. Thanks!

A Philosopher

ANM, thanks to you too!

Amanda

If journal editors really do think that only a small circle of elite people are competent to referee papers (I think this is very not true, but put that aside) then they should at least be honest and say it is peer review, not blind peer review. That's the problem.

Also, the idea of blind review is to *find* what work is really best. I guess some philosophers think that the fact someone went to a fancy school is better evidence than blind review. Okay, then that debate can be had. But given all the evidence that prestige is very strongly influenced by how well you did in high school, I don't understand how those who generally believe in prestige as a protector for quality (and I do, personally, think it is some proctor) cannot also admit there must be many, many, exceptions. And if there are many, many, exceptions, then a system focused on a small circle of "the best" is going to miss a lot of the non-traditional best.

Nicolas

Honest question: if peer review isn’t blind, what is publishing by non-elite folks in top journals evidence of? On the hypothesis defended on this thread, compromised anonymity primarily boosts the chances of elite folks. Since that’s a zero-sum game the bias in their favor works at the expense of others. But most folks are unknown to the elites (almost by definition) so most people benefit from blind reviewing. Is it argued that elite reviewers are biased against those they don’t know or recognize? If not, what is success by non-elites evidence of?

Anyway, just curious how this all pans out if, as we know, many elites struggle to publish and many non-elites don’t struggle to do so (you can’t count someone as elite *just* because they publish well or you’re sneaking your conclusion in to the analysis.) Again, not questioning the existence of egregious violations but I still see tons of people I’ve never heard of publish in the very best journals.

Humanati

Nicolas: I don't think we need to suppose that elite reviewers are biased against those whom they don't know or recognise, as opposed to neutral, say. (And I don't think anyone argued for this in the above thread.) I think the following suffices as cause for concern: being among the elite gives one *an advantage* in the peer-review process, since one is *more likely* to be reviewed by others who are biased in one's favour/to get a helping hand from (assoc.) editors, etc. You're right that those among the elite struggle with publishing too, since not everyone who serves as a referee is *guaranteed* to mix in the same circles. You're also right that those who are not among the elite succeed in publishing, since a neutral referee may very well like the paper and recommended it for publication. However, both of these observations are consistent with elites having an advantage. In response to your question, then, I think we should probably infer the following when non-elite folks publish in top journals: it gives us better (which is not to say perfect or even very strong!) evidence that the paper met the relevant standards than the evidence given by an elite person's publishing in that same journal.

Nicolas

Alright that’s helpful. If the referee pool is indeed small and skewed (of which I’m not sure we have good evidence but let’s grant it) then I see how some people may have an advantage in the process. And your response about evidence sounds plausible. My worry really is what we mean by elites, cool kids, too, etc. If we just mean accomplished, then, well I don’t see a concern. If instead the pooling tracks prestige over competence (regardless of the correlation), then I can see the concern, although A Non Mouse made a compelling argument for relying on such heuristics above.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)

Job-market reporting thread

Current Job-Market Discussion Thread

Writing Service


Categories