What's Wrong With Rankings? 2: Some Decisions Should Be Hard

By Sam Duncan

As I said in my last post I think the APDA is a much better place for those considering graduate school in philosophy to start than is the PGR, but I think using the APDA as a ranking would be a bad idea and I hope Jennings and others involved will never try to make it into one. Now I won’t deny that I think the PGR is especially problematic for a number of reasons, but I have larger issues with all rankings.  Relying on a ranking as a ranking to make a choice like where to go to graduate school is a very bad idea. To paraphrase Kierkegaard here rankings try to make what should be a difficult decision too easy.

To see why let me start with a ranking I cooked up by pulling from the most recent APDA’s ten year placement rates:

1. Massachusetts Institute of Technology (84%)

2.  Yale University (77%)

3. Baylor University (71%)

4-5. University of Tennessee/Pennsylvania State University (tied 67%)

6. Vanderbilt University (65%)

7. Rutgers (64%)

8. University of Virginia (63%)

9. University of Hawai’i at Manoa (58%)

10. UCLA (55%)

11. Villanova University (52%)

12-14. University of Chicago/University of Miami/University of Memphis (tied 50%)

15. Duke University (45%)

16. The University of Utah (42%)

No doubt you’ll notice there are a lot of things wrong with it (which is the point). For one the small differences in the placement rates between Rutgers, Tennessee, Penn State, Vanderbilt, and UVA are almost certainly statistical noise as is the similarly tiny difference between Villanova and the University of Chicago, the University of Memphis and the University of Miami.

More importantly, there is of course a huge difference between the kinds of jobs these programs place their students in. Tennessee, Rutgers, and UVA grads all have relatively good job prospects (though keep in mind relatively good is only a bit better than a coin flip) but the graduates who get jobs are going to get different kinds of jobs. To my knowledge UVA hasn’t placed a graduate in a research focused job straight out of the PhD program in the last ten years nor has Tennessee. Many Rutgers graduates who get jobs will get research focused jobs though. Or compare UVA and the University of Tennessee with UCLA and the University of Chicago. The evidence we have suggests that a graduate of the former programs has a statistically significantly better shot at permanent academic placement than does one from the latter. But University of Chicago and UCLA graduates are competitive for research focused jobs while UVA and University of Tennessee graduates usually aren’t.

If all you care about is getting an academic job at all then UVA and Tennessee win. If you only care about a job at a research focused school then UVA and Tennessee are out while UChicago and UCLA are decent enough options. However, what if you prefer a job at a research focused school if you can find it, but think you could be happy at a teaching focused school? Are the somewhat better chances of getting an academic job coming out of the University of Tennessee or UVA enough to outweigh the fact you’d be more or less giving up on a research focused job? The answer to that question is going to vary enormously from person to person. I also suspect that many potential graduate students may underestimate how happy they would be with a teaching focused job and instead defer to an unstated and undefended prejudice that such jobs are not good ones. I say this from personal experience as I wouldn’t have thought I’d like teaching at a community college when I was applying to graduate schools, but I’m honestly quite happy with it.

Or consider Baylor. It has an awesome placement record, but it also has a religious identity that it takes quite seriously. Among other things that has a big impact on the jobs graduates of its program get, which are often at schools that have not just religious identities but theologically and politically conservative ones. A lot of potential graduate students would not be happy with a job at Hillsdale College or Grove City College, then again there are people for whom either might be a good fit.

And what are you interested in and how strongly? Are you really wedded to one area of philosophy and wouldn’t be satisfied researching or teaching anything else? Are you particularly interested in one area but think it likely your interests will change? Do you just like philosophy and haven’t picked an area to focus on?

How you answer these questions will also affect your own personal rankings. If you are in the not sure category, then it makes sense to pick a bigger department where faculty have more diverse interests. Despite its incredible placement record even MIT might not be the best fit for you and you might want to look at somewhere where faculty are more eclectic in their interests. Or suppose you want to work in Chinese philosophy and wouldn’t be satisfied with anything else, then Hawai’i, Utah, and Duke may be the only entries here you should even think about. But what if American Pragmatism is your favorite thing in philosophy but can imagine you’d be happy working on and teaching something else? Should you go to Miami where you could work on that or to say a program with better placement but with no faculty working in pragmatism? I have no idea, but I do know that simply looking at where all your prospects were on my little list or any other like it would be a terrible way to make this decision.

On top of that there are a host of other factors to consider. For one thing, what are your career prospects if you don’t get an academic job? That should count for something but how do you weigh it assuming you even have the data? And good luck getting that data. Of the schools I’ve glanced at only Rutgers seems to be doing a good job listing non-academic placements. It's probably no coincidence that they also seem to be doing a good job placing students in good jobs outside academia.

There are important things to consider besides career prospects. No matter whether you get a job or not and no matter where you get a job if you do, the PhD program itself is going to take up a decent chunk of your life. Probably 4 years at minimum and quite probably more like 7. The quality of those years matters. What is the environment of the PhD program like? If you don’t get a job will you still look back on it fondly as a good experience? Where is the school located and how well will living there suit you? Here too there is no one size fits all answer. If you prefer living in a major metropolitan area and like being able to walk to things, Knoxville is not going to fit you. On the other hand if you loathe winter and like hiking in the mountains it beats Chicago, Boston, or the Philadelphia suburbs hands down.

Deciding whether to go to graduate school and where to go should both be difficult decisions that require one to think a lot about what one really wants and values. If you do it right, there is no one size fits all answer to this question nor any ranking that can tell you what you should do. You have to decide what matters to you. If you rely on rankings, whether mine, the PGR’s or another you are letting someone else make that decision for you. That might be a good enough way to decide what ink pen or coffee maker to buy. It’s not a good way to make a decision about what you want to do with your life. I’ll stop here for now, but in my next post I’ll have more to say about the value judgments that inevitably sneak into rankings

Comments

[concerned citizen]

Sam
I think you may be framing the issue in an incorrect way. Any young person considering graduate school in philosophy should attend to both (i) the Philosophical Gourmet, and (ii) the APA data ... and (n) ...
I doubt many slavishly follow one ranking in making their decision to apply to graduate schools. People need to consider (i) the location, (ii) their lifestyle, (iii) their dependence on family, etc. These rankings are but one source of data one should use in making one's decisions. But it seems undeniable that the Phil Gourmet is a useful starting point for choosing programs on the basis of specialties - I know the philosophy of science ranking is very useful.

[worried citizen]

Sam, have you heard of the straw man fallacy? You're arguing against a position surely no sane person in the field holds. No one thinks you should only attend to rankings when deciding where to attend graduate school. Have you heard people defending that position? Who are they? Where are they? I'd love to meet them!

[caleb]

I agree with the first comment, job placement numbers should only be one piece of data someone considers when deciding which programs to apply to. First, we're working with *very* small sample sizes. Most graduate cohorts are 5-6 people. This means that, if several people in a cohort decide that they'd rather pursue a job in industry, it completely messes up a program's placement numbers. (You do acknowledge this, but I think it's a more serious problem for your argument than you let on.) Second, people want different things
out of a career in philosophy. Some like teaching best, some like research best. If you fall into the latter camp, going to a school with a strong record of placing students into teaching jobs probably isn't a good idea. However, I think you do give one really good piece of advice. If your interested in studying something fairly niche like Chinese philosophy or philosophy of biology, then going to a program like Hawaii or Utah, respectively, will almost certainly be more productive than going to a Leiter T10 program where there's maybe only 1 person working in that area.

[Hrm]

I’ve not met anyone who decided where to go to grad school solely based on rankings without considering things like weather, plan B alternatives, ethos fit, research priorities, childcare, cost of living, fun, etc. I also don’t know that anybody involved in making rankings thinks they should be used as anything other than a partial tool.

That said, I think rankings (including PGR)*do* have a role as a partial tool. It makes sense for aspiring philosophers to want to know which school has the best reputation among their peers. Like the other aspects you name, this seems like a feature it’s quite reasonable to take into account.

[Michel]

worried citizen: I have seen many, many applicants themselves take exactly that approach. You used to be able to find them in the relevant sections of LiveJournal and TheGradCafe, but the first of those sites died years ago, and the other seems to have more or less died in the last few years. (I followed these sites for over a decade, incidentally.)

Some grads there would sometimes pay lip service to other factors, like placement or even specialty rankings, when they were in the process of deciding where to send their applications, but it looked to me like that was all it was most of the time. The dominant strategy was to just send applications to the first 20 (US-) ranked departments, and throw in three or four others based on specialty rankings, a well-known figure there, or just as MA-granting options. Come decision time, ordinal rank was pretty much all that mattered, apart from choices between a few programs at the very top of that list.

Once acceptances were in, it looked to me like applicants would sort into a couple of piles: those admitted to relatively high-ranking programs, who were confirmed in their ordinal ranking, and those admitted to mid- or low-ranking programs, who embraced their erstwhile lip service to other factors in order to make the best of things.

A few people (like me) stuck around over the years. Among that population, you could see different factors coming to the fore over time. Among them, I wouldn't say that any thought that one should attend only to the rankings. But really, by then it's kind of too late. For example, it took years and years of past applicants saying the same thing over and over before new applicants started thinking seriously about (1) their statement of purpose, and (2) their fit with the programs they applied to. That didn't change the application or acceptance strategies at all, but it did ultimately make a difference for how statements of purpose were crafted.

[Sam Duncan]

worried citizen,

I'm sorry what you say is just false. Like Michel I've known multiple people who thought like that. I remember having a completely exasperating conversation with someone about ten years ago who planned to go a program in a city they'd already decided they loathed even though they had an option to go to a program that was in much more pleasant one because the former was ranked several spots higher on the PGR. I also knew at least three people in my grad program who picked that program because it was ranked highly in one of the specialty rankings. (The kicker here is that two of them didn't end up writing their dissertations in those areas).
Do fewer people make that mistake now? God I hope so. I'd really like it to be the case that I'm arguing against a straw man. But I suspect I'm not. It may be that nobody will admit to making a decision like that today, especially if the decision blows up in their face. Then again no would admit to watching Heehaw and somehow it stayed on the air for 20 years.

[it’s alive]

I’ll add another voice in support of @Michel and @Sam Duncan. This absolutely does not seem to me to be a strawperson argument, but well represents the perspective of many to whom I talk.

The potential grad students I work with during application processes are fixated on the PGR rankings to the point of neurotic obsession, often in self-undermining ways. I’m thinking of people who want to do history of philosophy but still obsess over Rutgers because it is well-ranked, though history-free; those who obsess over NYU despite not really wanting to work in any of their supported subfields, etc. I do everything I can to get them to consider other factors, explaining that rankings correlate more to the reputation of a few well-known faculty members than the overall strengths and vibes of departments, and half the time I get the sense that they haven’t heard a word I’ve said and still just want to go to NYU or Rutgers (and into which they never actually get).

So FWIW, the inferred target of the argument here seems to me not to be made of straw, but instead very much alive and formidable.

[hrm]

@Sam Duncan this seems now to be a different claim. In the post the claim was that the decision should be hard, since it should involve weighing multiple things. Here it seems you think people should weigh things *in a particular way*. For instance, it sounds like you think it was simply an error for the first person you mention to privilege ranking over geography.

Why is this so? It sounds to me like they decided that reputation among peers mattered to them more than the city they would live in. People weigh lots of things in lots of ways, in lots of different careers. Some people choose worse job prospects for the sake of a spouse. Some make the opposite decision and live apart. Some people want to maximize their chances at earning potential, some want to maximize for felt enjoyment. People have different priorities.

I don't think it is up to us to say whether weighting rankings in a particular way against other considerations is a bad decision for a particular person.

(If you have a correct way of weighting geography, relationships, family, cost of living, maximizing career chances while preserving plan B, and enjoying oneself, I would love to hear it!)

[is the representation of the apda's placement data right?]

Was the University of Tennessee's numbers calculated properly? The APDA website has them as having had 10 permanent placements, 11 temporary, and 4 non academic placements.

I don't see any way for these numbers to together amount to a 67% placement rate. E.g. if you just divide permanent placements from the total number of graduates you get 10/25 = 40%.

MIT's most recent numbers also don't match the 84% figure given in the post. The APDA website has MIT as having had 24 permanent placements, 14 temporary placements, 7 non-academic placements and 2 unknown placements.

You can't get 84% from these numbers either.

Meanwhile, USC is absent from the list. But if you look at its most recent numbers and divide its permanent placement rate by the total number of people who graduate in the last ten years you get 67%, which would have placed it on the list.

[skeptical]

You write: "However, what if you prefer a job at a research focused school if you can find it, but think you could be happy at a teaching focused school? Are the somewhat better chances of getting an academic job coming out of the University of Tennessee or UVA enough to outweigh the fact you’d be more or less giving up on a research focused job?"

The APDA data doesn't support the claim that you have a better chance of getting an academic job coming out of Tennessee or UVA than you would coming out of Rutgers or UCLA. The APDA doesn't (and obviously shouldn't) factor in the kinds of jobs to which graduates apply. And the majority of people I know at top PGR programs are dead-set on fancy research jobs and haven't applied/don't plan to apply to the teaching-focused schools at which your hypothetical prospective student would be happy.

So why should this prospective student think that the data on placements will generalize to them? Given that they know that they're an outlier to a general trend, they really have no data about their chances of getting *any academic job whatsoever* coming out of a top PGR program.

[aargh]

the apda data is terrible. stop encouraging people to use it! I'm no fan of the PGR but I'd definitely prefer my students use it than the apda--a reputational survey where you know what is being measured (and how little it might apply to you), but where there aren't massive flaws/errors in the data is more valuable than a completely inaccurate placement survey.

[mit specificially]

MIT having an 84% placement rate is a straight up joke!

[can we forget about rankings?]

@hrm: I don't think it is an error of rationality on the individual level. I think that the ranking obsession, possibly rational on the individual level, is likely harming the discipline.

Apart from the affect on the quality of work we produce, there is a bad mechanism underlining that: students who think they are better researchers than others want to get into the more prestigious departments at the cost of other factors, and those who get into those are perceived as better researchers (or so they perceive themselves). If nothing else, it breeds more obnoxious people than is necessary (or average in society).

[target grad student]

several people higher up in this thread have said that this argument is a straw man. I can back up @Michel, @Sam Duncan, and @it's alive in confirming that I and many of my peers are still neurotically obsessed with rankings even after getting into good PhD programs :(

[Ranker]

I think Sam is right that graduate students tend to focus too much on ordinal PGR rankings (I certainly did). But if that was the point of the post, the title is a bit misleading. The phrase "What's Wrong With Rankings?" suggests that there's something harmful or misguided about program rankings as such. But this strikes me as the wrong attitude to take. The PGR does tell graduate students something important: it tells them how the quality of various graduate programs is perceived by leading scholars in the field. Surely, that's useful information, especially since there's very good evidence that academic hiring in philosophy is influenced by prestige bias (as I think another post on this blog pointed out some time ago). Moreover, graduate students aren't typically in an excellent position to judge program quality when applying for their PhDs, so it stands to reason that the PGR is valuable for them in the same way expert testimony is valuable in other contexts.

[david]

To Sam,
I admit to watching HeeHaw regularly. Perhaps if I had watched more of it, I would have had a more prestigious career in philosophy!

[still worried citizen]

I'm not sure that the fact that certain students misuse rankings in irrational ways much points to there being anything wrong with rankings. (My child had a habit of rubbing yogurt on her head, but that wasn't a mark against yogurt.)

[it's alive]

@still worried citizen, I think the problem is that the rankings are given to grad students who misuse them much in the way that your child doesn't know what to do with yogurt. So we can debate where exactly the problem lies (for instance, is it *in* the rankings or *in* the yogurt? maybe), but there is definitely a problem in any case. This problem is what worries Sam, and I think rightly so. The solution might not be to ban rankings (or yogurt), but some sort of solution is needed, hence the discussion.

[Additional data]

Classical logic counsels that if x is a comparatively better option than y, then x is a comparatively better option than y.