There are good rankings and bad rankings

There has been a lot of discussion on the philosophy blogosphere of late on whether the PGR should continue, and whether philosophy departments should be ranked at all. John Protevi, for example, has produced an "October Statement" soliciting signatories to endorse the idea of "No Rankings, Not Now, Not Ever."

I am actually very sympathetic with the idea that our discipline should do away with "reputational" rankings forever, and for a couple of reasons. First, I do not think there is any methodologically sound way to do reputational rankings. Reputation rankings must be based on either (1) the qualitative judgments of a select few (as the PGR does, in selecting its board/voting members), or (2) the qualitative judgments of the many (i.e. the philosophical community as a whole). Neither methodology seems to me sound in principle. (1) is  methodologically problematic because any selection of a select few will have to be peformed by other individuals with their own biases and philosophical preferences, and (2) is problematic because there is little reason to think that the many are good judges of the philosophical quality of departments across the discipline as a whole (as most of us are quite unfamiliar with the work of most other people). Second, I tend to think reputational rankings set up morally perverse incentive structures, ultimately concentrating power in the discipline in a relatively small number of individuals and departments.

So I, for one, am right with those who would like to do away with reputational rankings. But, for all that, I think it would be a mistake to do away with all rankings, particularly rankings on things like (i) job placement, (ii) attrition rates, and (perhaps) (iii) department climate. I believe rankings on these things are of great importance. Anyone who enters a PhD program should have access to sound information about whether the next 5-10 years of their life are likely to be a positive experience with good prospects for career success. Collecting and ranking graduate programs in these areas would not only provide prospective students with much-needed information; it would put much-needed outside pressure on programs to improve their performance--something I believe our profession has every right to expect of them. Too many graduate students over the years have suffered, and continue to suffer, from finding themselves in departments with poor placement rates, attrition rates, and climates. Quantitative rankings would hold departments accountable for their performance, and such accountability can be expected to systematically lead, in time, to better treatment of graduate students throughout the profession.

Or so say I. What say you, my fellow Cocooners?

Comments

[Alex Guerrero]

I don't see how you will get "quantitative" rankings of departmental climates that are any different than the "qualitative" rankings you decry.

There seem to be a lot of concerns about ranking departmental climates, or including that as part of some ranking, including the facts that people only experience one or two climates, people often have very poor information even about the full climate of which they are a part, there may be more than one "climate" at any particular department (depending on who one is, what one works on, what groups one belongs to, etc.), there are very serious small-sample size problems (things have been good for 3 years =/= things are good!), and there may be complicating factors favoring both overly positive (heads in the sand) and overly negative (on everyone's mind, lots of discussion and activity) reports.

These are all things that I've seen other people express in the previous thread, but I do think they are relevant to the question of how to present this information about "climate," and whether to include it in some kind of "ranking," even if it is worth gathering it (something I'm not at all sure about, given the above methodological difficulties).

So, you can put a number to it, but, as I'm sure you know (given your concerns in the post), that doesn't make it meaningfully "quantitative" rather than "qualitative."

[Marcus Arvan]

Hi Alex: Thanks for your comment. Those are all good concerns, and ones that may well make a "climate" ranking inherently problematic. But, for all that, the same concerns don't apply to ranking departments in terms of placement and attrition rates, do they?

[Ted Shear]

Here's my concern with the call to eliminate qualitative rankings: it won't stop individuals from making their own personal comparative quality judgments and reporting them to students. Sure, in an ideal world (where no one sits around and does that) the profession might be best served by a list of programs that have active researchers in different specializations that includes no rankings. But, that's not our world. People will make those judgments and report them to their undergraduates whether or not there's a codified ranking system.

You are absolutely right though about the methodological issues with any qualitative ranking system. Those problems are even worse though when the judgments of quality are made by individuals without any checks in place. I think that an important benefit of having a codified ranking system comes with the opportunity that it provides to specifically attend to those problematic issues and *try* to resist the biases at play (I emphasize 'try' because any attempt will almost certainly fail). Even if this doesn't actually work in combating the problems, the publication of a system of rankings affords us the chance to explicitly point out those problems and hopefully help people understand the ways in which the rankings may have been adulterated. This is something that I think has not been done well enough in the PGR, but has been attended to in some of the alternatives that others are working on.

I think this last point is really the strongest point in favor of maintaining rankings: when we have some formal system of ranking and publish it, we give ourselves the opportunity to make explicit disclaimers and point out just these issues.

[Marcus Arvan]

Hi Ted: I agree with your first set of points. People will always rank programs. My suggestion, though, is that it is better for people to rank programs *informally*, in a decentralized way, rather than through formal "reports."

Decentralized rankings--each person having their own judgments--can be expected to involve great diversity. I may rank things differently than you, etc. Individuals ranking departments informally isn't necessarily bad.

What I think *is* problematic are formal reports where some in-group's rankings/philosophical preferences have sway over the discipline as a whole, codifying that group's rankings/preferences as more or less definitive throughout the discipline.

[Ted Shear]

Hi Marcus, thanks for the reply! I think you've put your finger right on where we seem to disagree. You suggest, "it is better for people to rank programs *informally*, in a decentralized way, rather than go through formal reports." My concern is that if we don't employ a straightforward and transparent methodology for rankings, then its very likely that the responses of a large number of people will be deeply tainted with the implicit biases that we all know and love... errr hate. If we have a codified system of ranking, then sure they'll be susceptible to those biases (and other problems that you note and I'll get to), but at least we'll have the opportunity to try to actively set things up to avoid falling victim to them and (if nothing else) explicitly point out that they are surely confounding factors.

You're totally right that comparatively a single ranking aggregated from the reports of various individuals will express less diversity than the total set of individual's rankings. That said, this won't matter for those who are in most need of a metric for evaluation (i.e. undergraduates who come from smaller departments). I suspect that in many cases, those people will be in much worse shape.

But ultimately, I entirely agree with your last point that what is problematic is any formal report "where some in-group's rankings/philosophical preferences have sway over the discipline as a whole, codifying that group's rankings/preferences as more or less definitive throughout the discipline". I do think that this is a really important thing that we need to work to protect against, but it doesn't speak against having a codified system of ranking. This was very obviously an egregious deficiency of the PGR and hopefully whatever replaces it (assuming that something does) acknowledges this concern and tries to protect against it.

[Marcus Arvan]

Hi Ted: Thanks for your reply. I have to confess that I find the argument pretty compelling, but I still wonder whether formal reports will do a better job of counteracting implicit bias than a more decentralized alternative.