What's Wrong With Rankings 3: Hidden Value Judgments

By Sam Duncan

The thing I object to most strongly in rankings is that I think they all have value judgments baked in and that by using them one is accepting those judgments without thinking about or taking responsibility for a value judgment. Now I won’t lie, I find the values baked into the PGR especially bad ones, but I sincerely believe that this is a problem inherent in any attempt to rank graduate programs in philosophy or education more generally. Consider the way that judgments about what is good philosophy are hidden in the PGR’s supposedly objective and factual ranking. 

The way that specialty rankings are listed and what specialty rankings are listed is especially revealing. Metaethics and moral psychology get their own slot in the “Value Theory” heading and so does philosophy of law. But bioethics, digital ethics, business ethics, and environmental ethics are all rolled into one big ball and listed as “Applied Ethics.” It could be worse for them though. Feminism isn’t even judged to be “Value Theory” but is instead relegated to “Other.” Kant gets an entire slot to himself under “History” while Chinese philosophy and Indian philosophy are also crammed into the “Other” drawer. Indian philosophy doesn’t even get its own proper category in “Other.” Islamic philosophy isn’t listed anywhere. I don’t think anyone could accuse me of lacking sufficient respect for Kant, but that he gets a slot all his own while Zhuangzi, Confucius, Mengzi, and Lao Tzu are all stuffed into a single category that isn’t even counted as proper history of philosophy is indefensible. That the PGR tacitly says that al-Ghazali and ibn-Rushd aren’t even philosophers is so off base I’m literally at a loss for words.

What message does this send to a potential graduate student interested in say Zhuangzi or al-Ghazali? Pretty obviously that what they are interested in is less valuable than Kant or even isn’t real philosophy. These issues become even worse when one thinks about the general rankings and not just the specialties. There is a widespread perception that hires in the so-called “core” areas are more important for the PGR ranking. This too sends a message to graduate students about what is and isn’t good philosophy.

What is much worse is that administrators, especially at the sorts of institutions that tend to have PhD programs care very much about rankings like the PGR. Who then would have an advantage at a school that wants to move up the rankings, a brilliant bioethicist doing groundbreaking work on resource allocation or a clever and well-pedigreed, but ultimately unexceptional, LEMMing? I think we can already guess how a specialist in Islamic or Chinese philosophy does in this competition.

This is bad on a few levels. For one thing, there are a decent number of jobs in fields like digital ethics, environmental ethics, and feminism these days, and, at the moment relatively few new PhDs in those fields. Any school that really cared about placing its graduates could almost certainly do a good job at that by developing strength in those areas, and unlike moving up to the top 20 of the PGR from an unranked position or even scoring a spot on one of the LEMM subrankings it’s doable on a fairly modest budget. But that would not move the school up the PGR rankings. Administrators and faculty who care about that would oppose this move even though it would be very good for their future graduate students. I’ll add that this is not a hypothetical worry as I personally know of at least one PhD program that more or less trashed a very good applied ethics program with an excellent placement record in an attempt to make it into the PGR rankings.

All of this is just bad for academic philosophy not just as a field of intellectual inquiry but as a financially viable program. Bioethics classes tend to be in demand at most colleges and that is even more true of classes in the ethics of technology/AI and environmental ethics. If philosophy programs don’t hire people who can teach these classes they are leaving money on the table. Consider too that the number of college students is projected to shrink– perhaps quite dramatically– while at the same time that the students who do go to college are projected to become ever more diverse in terms of race and religion. We are reaching a point where we quite literally cannot afford prejudices of the sort that dismiss Islamic philosophy as not real philosophy and shove off Chinese philosophy into a “miscellaneous” category.

I know what the reply here will be: Administrators and faculty who use the PGR in this way are misusing it. I honestly can’t take this reply seriously and quite honestly have a hard time even seeing it as being offered in good faith. For one thing, it seems to be the most common blanket reply when anyone points out a potential way that the PGR could harm would-be graduate students or the profession as a whole. Worse, it’s exactly the same sort of reply that the NRA, casinos, or the alcohol industry give when anyone points out that their products do harm. “Hey we put labels on every single bottle of Thundertrain telling folks to drink responsibly. Not only that those labels specifically say not to drink and drive or drink while pregnant. It’s not our fault if people don’t use our product responsibly!” And as with alcohol and gambling it is very much in the interests of those who create rankings or do the ranking if their products are “misused” in this way. Problem drinkers account for most sales of alcohol and the most profitable group for casinos are always those with gambling problems. In the same way rankings have much more cache if faculty and administrators are using them to make hiring decisions or decisions about the entire direction of programs than they do if individual graduate students are using them to make decisions about picking programs.

Now I think the APDA is far preferable to the PGR in that it doesn’t hide value judgments in quite the same way (though as I said in the last post using it as a ranking would lead potential graduate students to make value judgments carelessly). And unlike ranking for perceived prestige, ranking programs for placement would incentivize decisions about hiring and department direction by administrators and faculty that would be positive for both potential graduate students and the profession as a whole. However, even using the APDA as a ranking would inevitably lead to abuses. This is a point that I think deserves a separate post.

Comments

[sahpa]

You object to rankings because by "using them one is accepting those judgments without thinking about or taking responsibility for a value judgment". Why should this be true?

Moreover, even when it is true, it isn't clear why that is objectionable per se. The value of thinking for oneself has to be qualified in a world of specialization and division of cognitive labor. Alas, I know lots of philosophers suddenly get very libertarian-sounding when it comes to value judgments in particular, valorizing some kind of unrealistic and undesirable self sufficiency.

You also lament how the PGR gets used by administration and so changes the incentives departments face. Fair enough, perhaps the PGR isn't the best tool we should be presenting to them to use. But some such tool is needed. No university administration would accept the *total absence* of any ranking scheme whatsoever, because such things are how specialized departments become legible to them qua outsiders.

You suggest toward the end that the APDA would be a superior ranking to provide (its "inherent problems" qua ranking notwithstanding) even for administrative purposes. This strikes me as naive, I'm sorry to say, precisely because administration at research-intensive universities, with philosophy researchers and a PhD program, do not only care about grad student job placement and grad student experience. They care, inevitably, also about *research productivity and prestige*. A ranking that is explicitly and directly about grad student placement is going to strike them as incomplete. Of course, the PGR is incomplete in other ways. It just isn't obvious why we should buy that the APDA is superior "for both potential graduate students and the profession as a whole" along this dimension.

[Michel]

I take it that the idea is that categories of the PGR are intended to reflect clusters of researchers to some extent. So the reason that Kant gets his own cluster, separate from other periods, is just that so many philosophers at research-oriented institutions specialize in Kant. This goes some way towards explaining why it's so hard for newly-popular or emerging subfields to get a foothold, since they'll have to build up significant concentrations of researchers at research institutions first. It's a bit chicken/egg, with a good dose of the Matthew effect. Now, I don't actually know what the criteria are for the subfield divisions, so I don't have a sense of how well they ultimately reflect that goal. My sense is that they (1) lag significantly, and (2) that there isn't a formal procedure to ensure a match between the categories and the sociological lay of the land, so this, I think, is where you concerns really gain traction, Sam.

That said, I'm not too bothered by the general division of subfields. What bothers me more is that the general rankings are kept separate from the specialty rankings. I'd have thought that a natural way of building the ranking would be to have it depend largely on the specialty rankings, so that the stronger a department is in more specialties, the higher its rank. At the top, then, you'd expect to see departments ranked 5 in many subfields, 4 in many more, etc., while at the bottom would be a pile of those whose modal specialty ranking is 1 or 2.

If the rankings were constructed this way, then the path to moving up the rankings would be clear: improve as many of your specialty ranks as possible. In specialties that are poorly represented at research institutions, like aesthetics, phil. of math, or Asian philosophy, this would be pretty easy--just hire a single well-regarded philosopher, or maybe two, and you could jump to the top of the specialty overnight. In more croweded subfields, it would take more, which would reward the profession for filling out neglected subfields, rather than trebling down on over-represented ones. At that point, of course, the way that the ranking breaks down specialties becomes much more important, and would need more careful attention.

Instead, the two rankings are separate entities and, despite the lip service paid to specialty rankings, my sense is that they're basically irrelevant (not least because most of us don't have a good sense of what the specialty rankings look like outside our own specialties). People just seem to default to the overall ranking.

(FWIW, I'm not a PGR skeptic, although I do think there's plenty of room for improvement in new and interesting directions. I think it's pretty useful, and I'm glad that we're a profession that maintains its own ranking system, rather than relying on useless external ones over which we have no control, like the THE or USNews rankings.)

[bleugh]

The NRA analogy is terrible. Assault rifles are designed to kill others, while the PGR is not designed to inform hiring or funding decisions. Nor, even, is it designed to settle application decisions. And alcohol and gambling are intentionally and knowingly addictive, undercutting the agency of their users. A better analogy would be something like kitchen knives: yes, some people misuse them and kill, but good luck making a sound argument from that fact to the conclusion that we should be seriously rethinking kitchen knives.

Endless students find the PGR useful and, in hindsight, are glad they used it. I used it ten years ago and I am glad I used it (and it didn't lead me astray). Why are you so insistent on ignoring the testimony of so many people who are glad the PGR exists, even if it isn't perfect?

[Andrew]

I didn't go to a PGR-ranked PhD program and don't teach at one now.

I think your diatribe has several flaws:
(1) People who gut a good applied ethics program are morons. And the institutional forces that push for that are mismanaging. If they are doing so by blindly imagining the PGR is an accreditation body, then it was merely a matter of time before they gutted it.
(2) The PGR succeeds in providing extremely important information that the average undergraduate student would not know that could inform them about a life-altering choice and its consequences.
(3) The supposed analogy to the NRA, gambling, alcohol seems really misguided. First off, it's difficult to see how they are a set, and your focus seems to be on the latter two was as predatory products that depend centrally on an off-label usage being its real meaning and purpose. It's not at all clear that the PGR secretly hopes that people abuse it harmfully.
(4) Instead, it makes some sense that what is informative for where you should go is also informative for where a lot of the good work is going on in mainstream philosophy.
(5) I don't actually think the ratings are negatively biased against Asian philosophy -- if anything this is a relatively employable area. But it is niche and excited undergraduates should understand that and not pick one of the few places that do it as if it's going to go well for them (the majority of people I know who went to such places are no longer in academic philosophy 5 or 10 years later).
(6) The ADPA is a superior tool in some respects and I'm glad it now exists. It probably only exists because the PGR exists and people realized that placement in jobs is an even better data set.

Again, more tools is better than less tools. Better tools are better than worse tools.

[Woodgrain]

This was helpful; I had forgotten that on the far preferable APDA ranking, SLU outranks the Ivy League by an aggregate 200 or so places.