Our books






Become a Fan

« Call for applications: Seminar on Intellectual Humility | Main | Only Naturalists Should Worry About Evolutionary Debunking Arguments »

10/04/2014

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Alex Guerrero

I don't see how you will get "quantitative" rankings of departmental climates that are any different than the "qualitative" rankings you decry.

There seem to be a lot of concerns about ranking departmental climates, or including that as part of some ranking, including the facts that people only experience one or two climates, people often have very poor information even about the full climate of which they are a part, there may be more than one "climate" at any particular department (depending on who one is, what one works on, what groups one belongs to, etc.), there are very serious small-sample size problems (things have been good for 3 years =/= things are good!), and there may be complicating factors favoring both overly positive (heads in the sand) and overly negative (on everyone's mind, lots of discussion and activity) reports.

These are all things that I've seen other people express in the previous thread, but I do think they are relevant to the question of how to present this information about "climate," and whether to include it in some kind of "ranking," even if it is worth gathering it (something I'm not at all sure about, given the above methodological difficulties).

So, you can put a number to it, but, as I'm sure you know (given your concerns in the post), that doesn't make it meaningfully "quantitative" rather than "qualitative."

Marcus Arvan

Hi Alex: Thanks for your comment. Those are all good concerns, and ones that may well make a "climate" ranking inherently problematic. But, for all that, the same concerns don't apply to ranking departments in terms of placement and attrition rates, do they?

Ted Shear

Here's my concern with the call to eliminate qualitative rankings: it won't stop individuals from making their own personal comparative quality judgments and reporting them to students. Sure, in an ideal world (where no one sits around and does that) the profession might be best served by a list of programs that have active researchers in different specializations that includes no rankings. But, that's not our world. People will make those judgments and report them to their undergraduates whether or not there's a codified ranking system.

You are absolutely right though about the methodological issues with any qualitative ranking system. Those problems are even worse though when the judgments of quality are made by individuals without any checks in place. I think that an important benefit of having a codified ranking system comes with the opportunity that it provides to specifically attend to those problematic issues and *try* to resist the biases at play (I emphasize 'try' because any attempt will almost certainly fail). Even if this doesn't actually work in combating the problems, the publication of a system of rankings affords us the chance to explicitly point out those problems and hopefully help people understand the ways in which the rankings may have been adulterated. This is something that I think has not been done well enough in the PGR, but has been attended to in some of the alternatives that others are working on.

I think this last point is really the strongest point in favor of maintaining rankings: when we have some formal system of ranking and publish it, we give ourselves the opportunity to make explicit disclaimers and point out just these issues.

Marcus Arvan

Hi Ted: I agree with your first set of points. People will always rank programs. My suggestion, though, is that it is better for people to rank programs *informally*, in a decentralized way, rather than through formal "reports."

Decentralized rankings--each person having their own judgments--can be expected to involve great diversity. I may rank things differently than you, etc. Individuals ranking departments informally isn't necessarily bad.

What I think *is* problematic are formal reports where some in-group's rankings/philosophical preferences have sway over the discipline as a whole, codifying that group's rankings/preferences as more or less definitive throughout the discipline.

Ted Shear

Hi Marcus, thanks for the reply! I think you've put your finger right on where we seem to disagree. You suggest, "it is better for people to rank programs *informally*, in a decentralized way, rather than go through formal reports." My concern is that if we don't employ a straightforward and transparent methodology for rankings, then its very likely that the responses of a large number of people will be deeply tainted with the implicit biases that we all know and love... errr hate. If we have a codified system of ranking, then sure they'll be susceptible to those biases (and other problems that you note and I'll get to), but at least we'll have the opportunity to try to actively set things up to avoid falling victim to them and (if nothing else) explicitly point out that they are surely confounding factors.

You're totally right that comparatively a single ranking aggregated from the reports of various individuals will express less diversity than the total set of individual's rankings. That said, this won't matter for those who are in most need of a metric for evaluation (i.e. undergraduates who come from smaller departments). I suspect that in many cases, those people will be in much worse shape.

But ultimately, I entirely agree with your last point that what is problematic is any formal report "where some in-group's rankings/philosophical preferences have sway over the discipline as a whole, codifying that group's rankings/preferences as more or less definitive throughout the discipline". I do think that this is a really important thing that we need to work to protect against, but it doesn't speak against having a codified system of ranking. This was very obviously an egregious deficiency of the PGR and hopefully whatever replaces it (assuming that something does) acknowledges this concern and tries to protect against it.

Marcus Arvan

Hi Ted: Thanks for your reply. I have to confess that I find the argument pretty compelling, but I still wonder whether formal reports will do a better job of counteracting implicit bias than a more decentralized alternative.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)

Job-market reporting thread

Current Job-Market Discussion Thread

Job ads crowdsourcing thread

Philosophers in Industry Directory

Open thread on hiring timelines

Cocoon Job-Market Mentoring Program

Categories