Reading recommendation letters

In our most recent "how can we help you?" thread, Amanda writes:

[W]hat are the ethics as far as *reading* [recommendation] letters are concerned[?]. Marcus, you said, you don't focus much on letters. Would a really bad letter turn you off a candidate you otherwise liked? I wills say that at R1's, and i've been involved in searches at 3 different institutions, letters are taken extremely seriously. Professors do distinguish between strong/stronger, sounding "restrained" etc. And with grad school applications they are maybe even more important. So what should a search committee member do when a candidate has, say, 1 really strong letter, 1 decent letter, and 1 bad letter?

Good questions. I'm really curious to hear how people answer. Here are my answers, beginning with a quick clarification. Amanda is right that I don't focus much on recommendation letters. This is because the science on this is unequivocal. First, recommendation letters as standardly evaluated poorly predict future performance. In fact, they have some of the lowest predictive validity of future performance of any commonly used measure. Second, they show demonstrable gender bias in academia specifically. Consequently, I'm with Mike Huemer: I don't think the hiring process in academia should include recommendation letters at all anymore. I think they do more harm than good, making hiring more a matter of personal reputation than actual achievement.

The only minor exception I have to this (and I may need to rethink it in the interest of consistency with the science) is that I have tended to give teaching letters some weight--but mostly in virtue of their descriptive (rather than evaluative) features. Basically, when I read a teaching letter, I read them to learn about what the person does in the classroom. What is their teaching style like? How do they engage with students? That's what I want to know, not what the letter writer thinks of their teaching (since I know very well from experience that different people can have very different views on what constitutes a good instructor or good classroom performance). When it comes to research letters, I have to confess that I give them very little weight, if any. If a person has published their tail off and I read their work and it is good and interesting, that's all I need to know. Conversely, if I a person hasn't published their tail off and I read their work and it seems mediocre, that's all I need to know. Either way, letters seem to me to introduce more noise than anything else. Someone can be a good philosopher but rub people the wrong way--or simply do a kind of philosophy (or do philosophy in a way) that their letter writers don't value.

I recognize that I am probably in the minority on this. Some people think letters are essential. While I disagree, I also recognize (despite the science) that there is substantial controversy over their usefulness. In any case, to answer Amanda's questions: No, a bad letter wouldn't turn me off. For all I know, a bad letter could be reflective of the candidate--but it could also be more reflective of the letter writer than anything else (I take it that I am not alone in realizing there are some meanspirited jerks in the profession, as in life more generally!). Now, if I got three letters, all of which raised very serious concerns about someone (indicating, let's say, that they behave unprofessionally), that would be a different story. But if I got one good letter, one okay letter, and one bad letter, I'd personally just set them aside and evaluate the candidate's file myself.

But these are just my answers to Amanda's questions. What are yours?

Comments

[Overseas TT]

I don't care about letters, at least about candidates in my own AOS, because (to put it bluntly) I think I just know my field well enough to trust my own judgment about candidates without having to know what other people think about them.

The one and only reason I take a look at letters is to see if there's any red flag about a candidate as a colleague - i.e. whether someone is difficult to work with or makes for a toxic work environment. This is very rare but I've seen/heard examples of it. Other than that, I couldn't care less about the letters, no matter who they are coming from.

[anonymous grumpy philosopher]

I have mixed feelings about letters, but when it comes to job candidates, very often no one on a hiring committee is a complete specialist in the area a candidate works in, and I think that while letters are often inflated and sometimes unreliable, they can provide helpful information from experts in a candidate's specialty.

Think about schools that use external letters for tenure; there, I take it the idea is (a) that people with significant expertise in a tenure candidate's subfield is going to be in a better position to evaluate that candidate's work than their colleagues may be and (b) by agreeing to write such letter, external writers agree to read the candidate's work carefully--not just one paper, but most or all of their work--and form a complete judgment about it--which the candidate's colleagues are (in most cases) unlikely to do.

I think both of these things translate to the hiring context. Someone's advisor, while obviously biased in certain ways (usually) in favor of the candidate, also has spent WAY more time reading the candidate's work, looking at drafts, having a complete picture of the candidate as a philosopher than the candidate's writing sample, cv, etc. can portray. So they are in a (b)-like position in comparison to the hiring department. Also, someone's advisor is way more likely to be much more of an expert in the thing the candidate is working on than the hiring committee is.

While I think there are many reasons to discount letters, I think ignoring these positive reasons in favor of taking them seriously amounts to a kind of intellectual arrogance. (You can of course think that these reasons are outweighed by the reasons against letters. But I think pretending they are not reasons is problematic.)

[Marcus Arvan]

grumpy: thanks for weighing in. Just to be clear: I don’t think we should ignore that there are some positive reasons in favor of taking letters seriously. I just think that the balance of evidence is that the negative reasons generally vastly outweigh the positive ones. Indeed, if epistemic humility is what you are after, then I think the best positive and most epistemically humble measure of someone’s research promise is their actual publication record. Why? If a person has published articles in journals X, Y, and Z, then that person has demonstrated that their work is good to multiple journal reviewers and editors who (due to anonymized review) are likely to be (A) specialists, who also (B) have no direct connection to the author and/or knowledge of who they are.

From an epistemic point-of-view, I’m inclined to think we’re much better off deferring to the judgments of multiple specialists in a peer review process designed to evaluate work on its merits over the judgments of 3 letter writers who have direct connections to the author—especially when we have independent scientific evidence that (C) letters of recommendations are poor predictors of future success, (D) tend to reflect particular kinds of biases, and (E) past publishing record appears to be the best predictor of future academic success. For studies on (E), see:

https://academic.oup.com/bioscience/article/63/10/817/238191

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6174868/

[Jonathan Ichikawa]

I think most letters are pretty irrelevant, but not all of them. I try not to pay any attention to how sincere someone sounds when they say the student is great, or to mere evaluations ("I thought her term paper was outstanding") as opposed to specific illustrations ("her paper brought out such-and-such point, which was a really useful innovation because..."). But letters can be useful when they contain actual information not found or highlighted elsewhere in the file.

For instance, a letter might be where I get the narrative to contextualise changes in the students' grades over their career. (This can come in personal statements too but it sometimes better from letter-writers who also know the situation firsthand.) If they talk about how the student took their course in their first year and struggled for some particular reason, then had a real moment where everything turned around for them, this will bear on how I understand low early grades in the transcript.

Letters are also informative when they tell me more specific things about CV items. What kinds of things did they do when they were your RA? What kind of material was covered in that logic class? This bears on what I should expect them to be familiar with already when they show up in my program.

[elisa freschi]

@Marcus, coming from continental Europe (where letters are either not used or less relevant and are anyway not (that) inflated), I am curious about the evidence you mention against letters.

I checked one of the articles you linked to (Aamodt et al 1993), which sounds quite interesting and sees letters as problematic especially when the author does not know the applicant well enough. Their conclusion is at the end positive (if developing further Peres and Garcia's method, etc.).

The other article (https://psycnet.apa.org/buy/1998-10661-006) is behind a paywall. My first question would be whether it is based on the US model of letters. Do you know? Thanks!

[Marcus Arvan]

Hey Elisa: thanks for chiming in, and good questions! The problem here, as I see it, is this. Very few people know about let alone use the Peres and Garcia method. A large plurality of studies have found that when letters are simply read—as is presumably done by philosophy search committees—they have little predictive validity. So yes, if people in philosophy started using the Peres and Garcia method, then letters could be helpful—and if they do start to use the method, I would be pretty happy! The problem, though, is that few (of any) people presumably use the method now, and few will presumably use it in the future (since by all accounts philosophers prefer to simply read letters to draw their own conclusions). This is a more general issue in hiring (which I learned from my spouse, who specializes in the area). Researchers like Peres and Garcia came up with their method nearly 60 years ago (!), yet so many decades later very few people use the method. For example, had you ever heard of it before? Do you know anyone who uses it? The problem then is that unless academic hiring committees start to use it (and they probably won't), then letters will remain the poor predictors for all intents and purposes.

I can’t re-access the second study while at home this weekend (I only have access to paywalled articles at work), but my recollection is that it was a meta-analysis of 80-something studies examining the predictive validity of letters in general, and the paper reports that they have among the lowest predictive value of any commonly used measure. I don’t recall whether the meta-analysis focused on the “US model” or not, but I don’t think it did!