Peer Disagreement & Peer Review

[cross-posted at Think: Just Do It!]

Roughly speaking, the problem of peer disagreement is the following:

You judge that p and an epistemic peer (i.e., someone who is just as competent and well-informed) judges that not-p. Should you stick to your guns and give more weight to your own judgment? Or should you give equal weight to both judgments, and so suspend judgment until further evidence comes to light?

On the one hand, sticking to your guns in the face of peer disagreement seems unreasonable and dogmatic. On the other hand, if giving equal weight applies across the board, then it seems that you should also suspend judgment about whether or not to give equal weight to incompatible judgments made by epistemic peers, which seems incoherent.

To sum up:

1. When A and B are epistemic peers, and A judges that p, whereas B judges that not-p, A (or B) can either stick to her guns or suspend judgment.
2. If A sticks to her guns, then that seems unreasonable and dogmatic (given that B is an epistemic peer).
3. If A suspends judgment, then A should also suspend judgment about whether to give equal weight to peers’ judgments, which seems incoherent.
4. (Therefore) A is in a position in which the options seem either unreasonable or incoherent.

Now, I think that the problem of peer disagreement raises interesting questions about peer review.

• Suppose that an epistemologist submits an epistemology paper to a journal that specializes in epistemology. The editor judges that the paper is worthy of consideration and sends it off to two referees (who are epistemologists, too). The reports come back with one “publish” verdict and another “reject” verdict. Should the editor give more weight to one verdict rather than another? If so, which one? Or should the editor give them equal weight? In which case, how would the editor decide whether to publish the paper or not?
• Similar questions can be raised even if the number of referees is odd. Suppose that an epistemologist submits an epistemology paper to a journal that specializes in epistemology. An epistemologist reviews the paper and decides on rejection. However, the author of the paper, who is an epistemic peer of the referee, obviously thinks that her paper is worthy of publication (otherwise, she wouldn’t have submitted it). So, should the editor give more weight to the author’s judgment or to the referee’s judgment? Or should the editor give the two judgments equal weight? If so, how would the editor decide whether to publish the paper or not?
• Similar questions arise at the level of the editor as well. Suppose that an epistemologist submits an epistemology paper to a journal that specializes in epistemology. The editor, who is an epistemologist, judges that the paper is not worthy of consideration. But the author obviously thinks that her paper is worthy of consideration (otherwise, she wouldn’t have submitted it). The editor and the author are epistemic peers. Should the editor give more weight to her own judgment? Or should she give equal weight to her judgment and the author's judgment, since the author is her epistemic peer? If so, how would the editor decide whether to consider the paper for publication or not?

By the way, a somewhat related set of questions arises with respect to expertise quite generally.

Comments

[Joe]

The scenarios you present with regards to referees and editors come up very often. I should know because I am an editor who is responsible for collecting referee reports and, ultimately, deciding whether a manuscript should be published in a future issue of the journal.

Practically speaking, the scenarios are relatively easy to reconcile:

• Case #1: The verdicts of two referees oftentimes disagree, one accepts the manuscript and the other rejects it. If the editor is fair, then it's likely a third referee will be invited to judge whether the manuscript should be published. The third referee's judgment breaks the tie.

• Case #2: The editor will likely trust the referee's decision because the referee should be impartial. (One concern I have about referees, though, is that more and more of them are rejecting manuscripts because they disagree with the thesis of the paper. That's not how they should judge papers. Papers should be judged by the argument it contains and whether the paper is of publishable quality, not whether the argument is something with which they disagree. Too many editors fail to make this distinction.) The manuscript's author is not impartial to the argument. Of course, she will argue that her paper is worth publishing.

• Case #3: If the editor decides not to send the manuscript out to external referees, it should be based upon the criterion I specified above, i.e., it's not "of publishable quality." If that's the basis of the editor's decision, then the editor shouldn't pursue publication of the manuscript. Sometimes authors are incorrect in their assessment of their own paper. Perhaps it is best that the author shelve the manuscript for a week or two before reading it again. Maybe putting it away and reading it again after a "cooling off" period will give the author a more impartial perspective of her manuscript.

I'm not sure the scenarios get to the heart of the problem of "peer" disagreement or peer review because editors are not necessarily peers. They have some authority over what gets published in their journal and what doesn't. Similarly, external referees are (supposed to be) impartial judges of manuscripts submitted to journals. If they take their role seriously, then they will execute their duties responsibly. Sometimes that is not the case, and editors should use caution determining the best fit of external referees.

The dilemma presented in the post is a difficult one to resolve, but I don't believe the scenarios accurately depict what's going on in the dilemma.

[Moti Mizrahi]

Thanks very much for your comment, Joe.

You remarks are very enlightening. From a practical point of view, editorial decisions need to be made regarding:

(a) Whether to send the paper to external referees or not;

(b) Whether to publish the paper (either with or without revisions) or not.

Now, I think you are right about Case 3. That is, it could be argued that the editor and the author are not epistemic peers as far as (a) is concerned. It could be argued that the editor of the journal is more knowledgeable about the needs and readership of the journal, and so the editor is in a better position to judge whether the paper is suitable for the journal or not.

This move, however, would not work in Cases 1 and 2, it seems to me, since referees and authors are epistemic peers as far as (b) is concerned. So, could you say a bit more about why you think that the scenarios do no accurately depict the dilemma sketched in the post?

As I see it, we can see how the problem of peer disagreement arises in the context of peer review by making the following substitutions in the dilemma:

Case 1: A = referee 1; B = referee 2 (the editor has to decide whether to give more weight to the verdict of referee 1 or the verdict of referee 2)

Case 2: A = referee; B = author (the editor has to decide whether to give more weight to the verdict of the referee or the judgment of the author)

From a normative, rather than a practical, point of view, I am not sure that deferring to a tie-breaker referee is the right way to make a decision in Case 1. Suppose that two referees say “reject” and one says “publish.” Why should we go with the majority? Is the reason supposed to be that the majority are more likely to be correct? Or less likely to be wrong?

As for Case 2, you rightly pointed out the problem with giving more weight to the judgments of referees. I think that the problem is even worse than simply referees rejecting papers with a thesis they disagree with. I think that it is rather difficult for referees to be impartial for the following reason. A common characteristic of philosophical debates is their bipolarity, e.g., rationalism/empiricism, internalism/externalism, physicalism/dualism, realism/antirealism, etc. Suppose an author submits a paper that argues in support of scientific realism. Her paper will be reviewed by referees with expertise in scientific realism, and so they will have a stake in the debate. Referees with realist inclinations are more likely to judge the paper favorably, whereas referees with antirealist inclinations are more likely to judge the paper harshly.

[Joe]

External referees and authors are epistemic peers but not necessarily in the case of reviewing manuscripts for publication. The author is not an impartial judge of his own work. External referees ought to be impartial judges of the quality of the manuscript. The external referee should judge whether the manuscript is worthy of publication (E.g., Is there an argument? Does the conclusion follow from its premises? Etc.). The external referee has an obligation to the journal and to the profession to insure that only high-quality manuscripts be published. So, because of the author's partiality and the external referee's responsibilities to the journal (and to the profession, more generally), I don't believe the two are necessarily epistemic peers, under the circumstances.

You ask: "Should the editor go with the majority? Is the reason supposed to be that the majority are more likely to be correct?" It's naive to believe that editors just go with whatever the majority decides. More likely (and at least in my case), the editor will carefully review the reports that external referees provide. If the two referees who reject the manuscript point out a *serious* flaw in the paper making it unpublishable, then the manuscript will not likely be published. Because editors cannot be experts in every sub-discipline, they depend on the views of experts in that sub-discipline to review a manuscript carefully. But, as an editor, one must read the reports carefully to determine whether the problems the referees raise in the report are serious enough that the manuscript not merit further consideration.

Likewise, if the external referee who recommended accepting the manuscript provided a report, the editor should review it carefully to better understand whether that report should be weighed more heavily than the reports that recommend rejection. There are lots of other scenarios an editor regularly confronts, and I haven't the space here for addressing all of them. Suffice it to say that good editors probably have a procedure for handling ties, partial external referees (and, yes, they're surprisingly easy to spot), etc.

Finally, you pointed out that it's difficult for referees to be impartial. I don't see why referees who are more inclined to agree with an author's thesis will judge a manuscript more favorably or why referees who disagree with the author's thesis will judge a manuscript more harshly. Competent referees do not judge a manuscript based upon whether they agree or disagree with the manuscript's argument; they judge it based upon whether the manuscript is worthy of publication. If it ever crosses a referee's mind that a manuscript ought not to be published because it is something with which they disagree, then I think they should re-think refereeing the manuscript.

[Moti Mizrahi]

Thanks for your reply, Joe. Once again, your remarks are very informative.

From a normative perspective, I agree that the editor is not an epistemic peer of the author as far as decisions of type (a) are concerned, since the editor knows the readership and the needs of the journal better than the author does. But that is not a reason to think that the editor and the author are not epistemic peers as far as decisions of type (b) are concerned. What is the reason to believe that the editor and the author are not epistemic peers as far as (b) is concerned? Presumably, the author is just as competent in evaluating arguments and as well-informed in the relevant literature (if not more so) as the editor is.

Similarly, if we believe that authors are not impartial, then we ought to believe that referees are also not impartial for the same reasons. That is, we think that authors are not impartial because they have a stake in the success of their own work. Similarly, referees have a stake in the success of work that bears on their own work. Since referees review work that bears on their own work, they cannot be impartial. This is not to say that they consciously judge papers either favorably or harshly. Rather, it is to say that referees’ decisions are influenced by unconscious biases. (Think of a philosopher who spent her career arguing that p, and is now reviewing a paper whose thesis is not-p. It would be very difficult, if not impossible, for this philosopher to be impartial because of unconscious biases.)

[Joe]

Let me make sure I'm still on the same page here because it's fairly easy to get lost in the dialectic on a blog. In the first paragraph of your most recent comment (7/21/12 @ 539p), you want to know why editors and authors aren't epistemic peers when it comes to editors deciding whether to publish a paper or not. (If I've got this wrong, I apologize. Let me know where I've gone wrong.)

First, I would not deny that editors and authors are epistemic peers if we're considering them independent of the circumstance in which the author submits a manuscript to a journal and the editor is an editor of the journal to which the author has submitted a manuscript. The author is, as you say, just as competent at evaluating arguments as the editor.

This relationship changes as soon as the editor assumes her role with the journal. The editor has a responsibility to the journal readers, journal reputation, etc., and is in a better position to judge whether a manuscript ought to be published in a future issue of the journal. The relationship between editors and authors is much like the relationship between assistant profs and chairs or assoc profs. Chairs and assoc profs have a responsibility to evaluate the performance of asst profs. They may be epistemic peers, but, because of their responsibility to the univ, the students, etc. and because of their position of seniority, chairs and assoc profs have an obligation to judge the performance of newer faculty members.

Next, your point of unconscious or implicit bias is a good one. I think it needs some careful consideration.

(1) I don't understand why referees cannot be impartial, at least when it comes to whether the article is of publishable quality. I believe good referees judge manuscripts based upon whether the manuscript is publishable, not whether they agree with the manuscript's thesis.

(2) Your response to (1) is that (a) the referee wouldn't want something published if it disagrees with what that referee has published or (b) there is some kind of unconscious bias affecting the referee to judge a manuscript more harshly.

With respect to (a), that kind of assessment seems to omit an important characteristic of philosophical discussion. Criticism of one's position is just as good as the endorsement of one's position. An author may have a stake in the success of their own work and that success might arise not just from colleagues endorsing the referee's position but criticizing it. So that a referee wouldn't want a well-written, well-argued manuscript criticizing the referee's own position would be unusual and suspicious (especially in the mind of the editor).

With respect to (b), I agree that the referee might resist endorsing the manuscript's thesis, even if she knows that the paper is well-written and well-argued, because of some unconscious bias. I think it's something the referee should try to overcome, especially if it's potentially the case that the referee's profile increases when the criticism is published. Because unconscious bias is not something we consciously track, the referee will not be able to overcome the unconscious bias. A referee who's affected by some kind of bias will submit a report that looks very different from other reports. Should the referee submit a report detailing criticisms of the author's position without citing deficiencies of the argument, I think editors can judge that it is some kind of psychological bias affecting the referee's verdict. Referee reports are quite revealing not only to authors but to editors as well.

[Marcus Arvan]

I'd like to add something to what Joe said. I don't think journal editors are our epistemic peers -- at least not most of the time. Becoming an editor at a respected peer review journal is almost always a very hard-earned honor. The person has usually made outstanding contributions to the field, and has demonstrated abilities to evaluate arguments that most of us who submit have not. For example, when I submit to the Journal of Moral Philosophy, I am well aware of the fact that Thom Brooks is a much more accomplished philosopher than I. He has earned a certain claim to epistemic advantages over me. This is not to derogate me. It is to say that a person's accomplishments and standing in the field are epistemically relevant. Just as it would be bizarre to think that a first year resident in medicine will tend to be anywhere near as good of a judge of diagnoses than his/her supervising physician, so too I think it is incorrect to suggest that most editors, most of the time aren't better judges of merit than we.

[Moti Mizrahi]

Thanks for the clarifications, Joe.

I think we’re on the same page (which is not to say that we agree).

As I understand it (and please correct me if I am wrong), your claim is that editors and authors are not epistemic peers when it comes to type (b) decisions, i.e., whether to publish a paper or not, because of their roles. That is, in the context of academic journals, there is a hierarchy, where the editor is at the top by virtue of her title or rank, and the author is at the bottom by virtue of her title or rank. In other words, editors and authors may be epistemic peers (by virtue of their reasoning skills), but they are not professional peers (by virtue of their ranking in the hierarchy). If that’s the case, however, then the following question comes to mind: How does one’s rank put one in a better epistemic position to make type (b) decisions? Does having more (or a certain kind of) professional duties put one in such a position?

Marcus’ comment could be construed as a reply to this question. That is, editors and authors are not epistemic peers, not merely because of their professional roles and ranks, but because of the superior academic accomplishments of editors. Marcus, could you please say a bit more about that? For example:

What if the editor and the author are equally accomplished?

What if the editor is less accomplished than the author?

How does being more academically accomplished (i.e., having more publications than a colleague) make one better at reasoning and evaluating arguments?

If I have four publications more than a colleague of mine, does that mean that my reasoning skills are better?

Your (Joe) remarks about editorial oversight are well-taken. It is very reassuring to read that editors exercise the kind of editorial oversight you describe over referee reports.

[Marcus Arvan]

Moti: I don't think it's the kind of thing one can quantify in terms of publications. I do think it's the case here, as in other areas of life, that a persona accomplishments are a strong but defensible measure of their being more competent than you. A person who has achieved the status of a high ranking journal editor has simply accomplished things in the profession that I (and most others) have not. They have demonstrated to other members of the profession that they have a certain high level of expertise. Their having accomplished those things means something.

Now I expect you will once again ask for clarification, but again, I do not think it is the sort of thing that can be clarified. It is a matter of what I would say is Aristotelian practical wisdom. Our entire discipline -- like other disciplines -- is a diffuse system of a division of labor. The system as a whole is designed, through peer review, tenure review, etc., to function as a kind of epistemic test. You can deny that the system works, but I would say it clearly does. Is Thom Brooks a better philosopher than everyone who submits to his journal? No, but I'm quite sure he's better than most. This is not merely because of his accomplishments but also because I've read his work. I *respect* him as my epistemic better both because he has accomplished stuff that I haven't and also because I just think his work is better. I also think that this will probably generally be the case with anyone who is appropriate humble about their own work. I think I'm a good philosopher, but I would be absolutely kidding myself if I were consider myself the epistemic peer of the editors of Phil Review, Nous, etc.

The division of labor here, as elsewhere, is profoundly imperfect, but I just don't think it is the least bit plausible that journal editors are (most of our) our epistemic peers. I probably won't be able to convince you. Perhaps we will just have to agree to disagree.

[Moti Mizrahi]

Thanks for the clarifications, Marcus.

Your claim seems very reasonable to me. But I think it can be understood in at least two ways:

1. Quantity of academic accomplishments is a defeasible predictor of reasoning skills.

2. Quality of academic accomplishments is a defeasible predictor of reasoning skills.

I think that either (1) or (2) might be true (even though it would be difficult to substantiate these claims; (2) more so than (1)), but only as far as specialty journals are concerned. What about general journals? Is the editor of a general journal also the epistemic superior of (most of) the authors who submit work to that journal? Presumably, the editor’s accomplishments will be in a rather narrow field relative to the wide range of fields from which submissions to his journals will be.

In addition, if either (1) or (2) are true, then it means that we cannot make the blanket statement that editors are the epistemic superiors of authors. This is so because:

1. Sometimes, the author is as accomplished as the editor.
2. Sometimes, the author and the editor are equally accomplished.
3. Sometimes, the author is just brilliant. [We need to take that into consideration, especially in non-empirical disciplines, such as math, logic, and (some of) philosophy.]
4. Sometimes, the editor may be incompetent despite his/her accomplishments.

[Marcus Arvan]

Moti: thanks for your thoughtful response. As far as generalist journals are concerned, that's why there's an editorial board. The members of the editorial board *are* specialists in the various areas the journal covers, and (as far as I am aware) they are typically the ones who consider the reviewers' comments and forward an overall recommendation to the Editor.

As for your more general comments, I certainly wouldn't make the blanket claim that Editors are our epistemic superiors. There surely are cases like those you list in (1)-(4) -- though, for what it is worth, I think editors tend to recognize when a paper is absolutely brilliant and there are few, if any, incompetent editors at well-respected journals. As for the first two cases (the author is more/equally accomplished as the editor), I think editors tend to recognize this too and respond appropriately to the fact that in these cases they are not epistemic superiors. It's a not-too-dirty secret, after all, that it becomes somewhat easier to publish as one gets better known, and that editors sometimes use less-than-fully-blind review with super-well-known people. If accomplishments are a fairly reliable guide to one's abilities as a philosopher -- and I think they are -- these practices make epistemic sense. [I expect many have a problem with these practices; I do not]

Finally, I prefer not to focus simply on the Editor. The Editor is but one link in a very large chain -- a division of labor explicitly designed to determine (albeit imperfectly) which pieces of work are better than which, and which are good enough to publish. I expect you may want to deny that the machine works as advertised, and in that case we can have a discussion about that. But, for my part, I think the machine does work.

In full disclosure, I used to think that I was, like, totally brilliant and my paper rejections were a result of small-minded people not seeing my brilliance. I have learned to see this attitude for what it was: youthful arrogance. I have learned that my papers did not get published because they in fact sucked, or at least weren't good enough yet. My experience, in other words, has taught me that the machine *does* work -- very imperfectly, to be sure, but a better judge of good and bad work than I or most other people are as individuals.

I'm belaboring this point because I think it is one of the more important things I've learned professionally. Seeing that it is not editors or reviewers who are (generally) incompetent but rather *me* who was not doing good enough work was, I think, an important, but hard fact to face -- and a crucial part of my maturation process as a philosopher.

I guess that's my closing thought. In my own case, I just find it more worthwhile to focus on what *I* could be doing better, seeing the system for what it is -- a good but imperfect measure of quality -- rather than worrying about incompetent editors and reviewers. As a general rule, I judge: if *I* think my paper is great and other people (including editors) think it sucks, it probably sucks; I am deluded. Maybe not always accurate, but a pretty good rule.

PS: I'm *very* sorry about how rambling my comments have been the past day or so. Just got back from a brutal fly-out in a land far, far away and am still suffering from coherence-sapping jet lag. ;)

[Moti Mizrahi]

Thanks for your rambling comments, Marcus :)

From a practical point of view, they make sense to me. But I do want to clarify that I am asking a *normative* question (How “the machine” ought to work? Not, How “the machine” does work? or How can I work with “the machine”?)

So, the machine may be working in roughly the way you described, but is that the ideal way? I don’t think so, because of the epistemic problems mentioned in the post.

You may be right that editors are generally competent and reviewers are generally unbiased, and so on. I don’t know. These are empirical questions. What I can say is that there are no obvious *epistemic* (as opposed to pragmatic) reasons to believe that editors and referees are in a better position to judge the argumentative quality of papers than authors are.

[Marcus Arvan]

I guess just don't see it. I think there *are* obvious reasons, just like there are obvious reasons NFL football players are better than collegiates. There is a division of labor where, if you make it to the next level, that is because you have demonstrated competence at a lower level. There are all sorts of mechanisms in place in academic philosophy to establish who has what levels of competence. One only gets to be an editor at a top journal by demonstrating very high levels of performance over a long period of time according to the standards of the profession. If that's not a reason to think that editors, on average, have better epistemic credentials than (most of) the rest of us, I don't know what is. I admit that like all things in this world, it is very imperfect -- but I think it is clearly better tha each person just judging for themselves. Most people think their own work is awesome. Most of them are deluded. The professional processes and standards that exist have numerous structures in place to ensure that those higher up in the chain are (on average) better judges of quality than those lower down. I also take the outcome of the procedures to be pretty darn good. The "big shots" in philosophy -- Tim Williamson, Elizabeth Anderson, I could go on all day -- really do better philosophy than most of the rest of us, and what do you know: the machine got it *right.* Sure, not *always* right, but more right than not, and more right than any alternative I'm aware of. There's my evidence. What's the evidence to the contrary? What alternative machine would do better?

[Marcus Arvan]

Actually, here's an answer to my last question. As I noted in an earlier post, I think the physicists have a better machine -- one that grapples with the issues you're raising, Moti. That machine, very roughly, consists in (A) an online repository and (B) widespread vetting and discussion on blogs, such that there is already community-wide agreement on which papers are good and bad before a paper even gets sent out for review. As I noted in that earlier post, I prefer that machine for roughly the reasons you raise. Public vetting by the profession at large is more transparent and less apt to be corrupted by bad luck (e.g. Incompetent reviewers) than our procedures in philosophy. Still I stand by the other stuff I wrote. I think there is plenty of good evidence that editors are in most cases our epistemic superiors.

[Moti Mizrahi]

Marcus,

I am glad we agree that philosophy’s peer review system is not ideal and that there are better systems out there. From reading your post, it seems to me that physics’ system is indeed better insofar as it is more egalitarian than philosophy’s system, which is basically a caste system.

As for editors being epistemic superiors, please correct me if I am wrong, but it seems to me that you are assuming that philosophy’s system is purely meritocratic (e.g., you write that “if you make it to the next level, that is because you have demonstrated competence at a lower level” and that “One only gets to be an editor at a top journal by demonstrating very high levels of performance over a long period of time”). That is, if one makes it to the top (i.e., becomes an editor) it is only because one has demonstrated superior philosophical skills, and thus deserves his/her place at the top. I’m not sure that’s the case. Again, these are empirical questions and I don’t have the empirical evidence. Given what we know about the peer review system in philosophy, however, there are reasons to doubt the “pure merit” assumption:

1. We can say with some confidence that most editors are the epistemic superiors of most authors only if we can say with some confidence that most editors merit their positions (on account of their superior philosophical skills).
2. We cannot say with confidence that most editors merit their positions, since the peer review system in philosophy is not designed to eliminate things like biases, “members only” clubs, political connections, financial advantages, pedigree, etc.
3. (Therefore) We cannot say with confidence that most editors are the epistemic superiors of most authors.

For example, suppose that Journal A is a members-only club. Because there is no transparency, most of us don’t know that. Now, a member of the club, who has published a few articles in Journal A, might be wrongly judged as more deserving of a place at the top than a philosopher who has published the same number of articles (perhaps even more), but in Journal B, which is thought to be less “prestigious” than Journal A.

[Marcus Arvan]

Moti: I deny your second premise. I think the peer-review system -- and the rest of the machine -- is designed to *mitigate* biases. Note I did not say "eliminate." And no, I never said *purely* meritocratic. No system can eliminate biases, no system is purely meritocratic. This is the problem I have had with your arguments in the thread. You seem to think that if a system has any faults, it is bad system. I simply see no evidence for the rather cynical view your second premise suggests. I think I can say with confidence that most editors merit their positions. I have given you evidence. We have a distributed system of a division of labor designed to *mitigate* bias. I also think it is relatively successful, and I have given you evidence that it is (e.g. the system *does* select better philosophers to publish in places like Phil Review, etc.). Where is your evidence for your second premise? Your remarks suggest that I must show that premise (2) is false. No, the evidential onus is on you -- the person giving the argument -- to show that it is true. I have seen no evidence, as of yet, that it is true. A good system need not *eliminate* bias. It must merely mitigate it, reasonably well, such that those in positions of authority have earned those positions meritocratically. Again, I have given evidence that they tend to. Where is the evidence that they haven't?

[Moti Mizrahi]

Marcus,

I am sorry if I misunderstood your argument. Your charge of cynicism and demand for empirical evidence give me the impression that you have misunderstood my argument as well. I am making a purely theoretical argument from a normative point of view.

So let me sum up the dialectic as I see it. Please correct me if I got it wrong.

I take it that your argument is an IBE:

(1) Highly accomplished philosophers occupy the highest ranks in the system, such as editorships, etc. [You take that as a given—a phenomenon to be explained.]
(2) The best explanation for (1) is that these philosophers merit their top ranks because they have superior skills (i.e., they are the epistemic superiors of the majority of other philosophers).
(3) No hypothesis can explain (1) as well as (2) does.
(4) (Therefore) Highly accomplished philosophers who occupy the highest ranks in the system merit their top ranks because they have superior philosophical skills.

In response, I pointed out that (3) is unwarranted, since we cannot rule out the following alternative hypotheses:

a. Philosophers at the top of the hierarchy are there because they are politically well-connected.

b. Philosophers at the top of the hierarchy are there because they are members of insiders clubs.

c. Philosophers at the top of the hierarchy are there simply because of pure luck.

d. Philosophers at the top of the hierarchy are there because of their pedigree.

To be clear, I am NOT saying that any of these alternative hypotheses is true. Rather, I am saying that these alternative hypotheses are plausible, which is enough to cast doubt on (3). In a system with no equality and no transparency, we cannot rule out these alternative hypotheses, and so we cannot be confident that (2) is true.

[Marcus Arvan]

Moti: I think by any ordinary standards of evidence, we can indeed rule out those things. I have given two lines of argument for our having this evidence: (1) the existence of a system of divided labor with many (albeit imperfect) safeguards to protect against all of those things, and (2) evidence that the system actually tends to select good philosophical work over bad. It seems to me that if your argument appeals to ordinary evidential standards, it fails for both of thes reasons. On the other hand, if you do not mean to appeal to anything like ordinary evidentiary standards -- if, for example, you mean to use infallibility standards (viz. I cannot rule out that I am a brain in a vat either) -- then I think the argument is sound but practically trivial. I can't infallibly rule out your four hypotheses any more than I can infallibly rule out being a brain in a vat, but so what? Ordinary evidential standards are clearly more relevant, and again, on these standards I think the argument fails.

[Moti Mizrahi]

Marcus,

I was under the impression that we agreed that an egalitarian system with transparency is better than a non-egalitarian system with no transparency. (Your post on the peer review system in physics suggests that.) If so, then we should also agree that, in a non-egalitarian system with no transparency, we don’t really know how editorial appointments are made, how referees are selected, how editorial decisions are made, etc. We only have rumors and anecdotes—but no real knowledge—because this sort of information is not publicly available in any systematic way. If that’s the case, then I don’t see how we can rule out, with at least *some* confidence, alternative hypotheses a-d.

From your last comment, I get the impression that you think that, even without transparency and equality, a system can be effective insofar as, for the most part, excellent researchers make it to the top. So perhaps you could say a bit more about those “safeguards [that] protect against [bias, favoritism, discrimination, preferentialism, etc.]” in a non-egalitarian system with no transparency? [This might actually be an idea for a follow-up post.]

[Marcus Arvan]

Moti: yes, I agree that a system with more transparency is probably better -- but your other impression is also correct. I think that we can and do have plenty of evidence that the present system, without transparency and equality, is a relatively effective system. My evidence, again, is:

(A) There are numerous structures in place (blind review, etc.) that can, due to their very nature, be reasonably expected to filter out a great deal of bias and place (relative) epistemic superiors in charge of decisionmaking.

and also,

(B) There is plenty of evidence that in fact these structures work relatively well (people like Williamson, Anderson, etc., publish in top journals; people like me do not; better work *tends* to appear in top journals; worse work does not; etc.)

So, yes, I think a system without transparency and equality can be relatively effective, as I think there is plenty of evidence that such a system *is* effective. This is all consistent with my other, more general claim that I think a more transparent system would be better.

[Moti Mizrahi]

Thanks for this excellent discussion, Marcus.

So let me see if I got this right.

To say that the system is effective is to say that, for the most part, the highly accomplished make it to the top.

Now, you argue that (A) and (B) are evidence for this effectiveness of the system. I’m not sure about that.

In (A), you mention blind review. How can blind review be evidence for the system’s effectiveness if blind review is the very thing that is not transparent? We don’t know to what extent reviews are blind. We don’t know what measures are taken to ensure that reviews are blind. This sort of information is not publicly available in any systematic way.

As for (B), perhaps being highly accomplished is coincidentally linked with being at the top. In other words, there might be just a coincidental correlation between accomplishments and ranking. But that doesn’t mean that those at the top are there *because* of their accomplishments. We need evidence of causation, not mere correlation. Again, we don’t have that kind of evidence because this sort of information is not publicly available in any systematic way.

Finally, if an egalitarian system with transparency is better than a non-egalitarian system with no transparency, do we (as a profession) have an obligation to try to change the system such that is becomes more egalitarian and transparent?

[Marcus Arvan]

Moti: I think we do have that obligation. But as for your claims about us not having evidence for the considerations I speak of, I deny that. There may not be rigorous scientific evidence, but then again, I don't have rigorous scientific of the bus routes around my city. I have converging anecdotal evidence from a number of sources. I know there is a bus system that sets routes. I know those routes are posted. I've seen busses come to different stops around the posted times. That is all good evidence. I believe we have similar evidence in the case we are discussing. There are some journals (I will not name which) that were suspected by many for not respecting blind review. I believe there is evidence of all sorts that many (most) journals use pretty decent bias-reducing practices, and that those that don't are pretty well known. And again, I think the evidence is strong and various, from many different sources. You can of course deny that this is the case, and I guess at that point we will have to agree to disagree. I think your skepticism is unwarranted given the nature of the evidence we face, you think it is warranted. Not sure what else to say, aside from the fact that if sound empirical studies could be developed and performed (which I doubt is possible), I would happily and confidently place a bet on my position. For again, I just see a ton of evidence for what you deny.