Tips from Search Committee Members: how do you evaluate recommendation letters?

The first six posts in this series examined how hiring committees read and evaluate CVs, cover letters, research statements, teaching statements, diversity statements, and writing samples. Before we move onto first-round interviews and various aspects of on-campus visits (e.g., job talks, teaching demos, etc.), I'd like to finish up this first part of the series by asking search committee members to share how they read recommendation letters.

I've heard anecdotally that some search committee members care a great deal about recommendations, whereas others care very little. The Cocoon has also fielded a lot of questions over the years about whether recommendation letters need to be updated each year, on how many letters to include (including teaching letters), on whether it looks strange to not have a letter from someone on their committee or from their current job (e.g., a postdoc or VAP), on how important it is to have outside letters from people not in your grad program, and so on.

So, I think this thread could be really helpful. If you're on a search committee or have served on one in the past:

1. How much value do you place in recommendation letters? A lot? A little? None? Why?
2. What type of institution do you work at? R1? SLAC? CC?
3. Do you notice or care whether a recommendation letter is updated or a bit old?
4. Do you notice or care about whether a candidate has letters from all of their committee members? Outside letters? From their current job (post grad school)?
5. How do you read letters?
6. What types of things reflect well on a candidate, and what kinds of things reflect not so well?
7. Anything else?

Curious to hear everyone's answers!

Comments

[Caligula's Goat]

I'll start with a mildly cynical take:

1. Almost none. The job market has turned letter writing into a sort of arms race. Letters tend to be entirely positive and so don't tell me anything specifically unique or interesting about the candidate. Giving letters a lot of weight, under these circumstances, really hurts candidates whose letter writers are honest.

People in my department are all over the place about letters. Some care a lot about them for all sorts of reasons (prestige of writer or program, research potential, and so on). I find the CV a better source.

2. A research-intensive SLAC (we require 6+ decent publications for tenure and a book or another 6+ for full)

3. I don't notice but again some people in my department notice and really care.

4. To the degree I notice at all, what I care about is that the candidate submitted all (and only) the number of letters we ask for. This is about fairness for me. If we ask for three letters, please give us three (not two or four) so that we can assess candidates equally.

5. Very quickly, if at all.

6. Tangible, concrete evidence. Someone talking about how much potential you have or how great you are in the classroom isn't worth much to me (almost all letters will say this). Letter writers need to show and not tell. Tell us what the candidate has actually done (win awards, run graduate conferences, designed new popular classes).

7. If at all possible, ask someone in your department (usually your DGS) to read through your letters. Some letters are actively harmful to your candidacy. These are rare but they exist.

[Bill Vanderburgh]

Caligula's Goat captured it perfectly. I (and my department generally) pay little attention to letters. At our large public R2, letters from "prestigious" writers tend to focus on research, which is a top-3 concern for us, but not top-2, and we feel we get a better (more objective, less "spun") assessment of it from the cv, writing sample, and interview.

Old letters are a bit of a red flag, but not much since we don't count them for much anyway. More important, we like to see a letter from someone at the current institution if the person has already graduated (to help us assess teaching and collegiality).

I don't notice if there isn't a letter from all committee members. In fact, if there are more than three letters, it greatly diminishes the chance that I will read any of them carefully.

[R1 faculty]

There's a lot of variation about this in my department (R1 with a PhD program). I take letters pretty seriously, but more in terms of gaining additional information about the content/importance of research and teaching than about assessment. However, I disagree with the common claim that all letters are just glowing and useless in terms of assessment. It actually seems to me that while the majority of letters are about the same level of extremely positive, there is a tier above that and also a tier below that, and being in those tiers provides helpful information. (Sometimes someone has letters that make clear that more than one faculty member at their grad institution, or more than one letter writer, thinks they are... a more strong candidate than anyone else typically coming out of that institution. And likewise sometimes someone has more than one kind of lukewarm letter. I probably wouldn't read much into one super over the top letter or one lukewarm letter, but if someone has four letters saying they are incredible, I'm definitely going to pay more attention to the rest of their file (it wouldn't trump, say, my reading of their writing sample, provided I think I am competent enough to assess it and think it has serious problems). And if someone has more than one lukewarm letter, I'm probably not going to bother unless I was really captivated by the rest of their application somehow (but I mean really very captivated!).

I definitely notice when letters are old but usually don't care if it's just one year (but it's better if they are not old!); I take outside letters for people still in grad school or near grad school more seriously than internal letters in terms of the actual assessment.

Basically the things that I find most helpful/positive in letters is someone's ability to say, in a different way than the candidate, what is valuable/interesting/novel/compelling/exciting about their research. As someone who has written a lot of letters, this is genuinely hard to do for certain candidates, and a way of reading between the lines about glowing assessments is: what is the actual thing that the letter writer thinks the candidate is contributing philosophically that is not just how the candidate is already presenting themself?

[SLACcer]

I could care less about recommendation letters. I don’t even read them. I don’t care what other people think of an applicant. The rest of their dossier tells me everything I need to know.

[JW]

We're almost always hiring in an area outside of my speciality. So yes, I read the letters of recommendation to use as extra information about the candidate and their research. I am happy to admit when I am not qualified to judge on the originality or significance of a paper. I also read the writing sample, and if I'm interested, other published papers or what I can find on their website. In these cases, as long as the letters are good, I place the most importance on how interesting and accessible the papers are (since I am hiring not just a researcher, but a colleague).

As to the list of questions: there are so many reasons the candidate could have gotten a letter from X instead of Y, or letter writers could have updated or not, that I try not to read into it.

FWIW, my colleagues in my research-active department have differing views on letters, from not reading them at all to using them in their first pass considerations.

[Tenured R2ish faculty member]

I literally don’t read them and don’t look at them. I wish we didn’t ask for them.

Here’s why: I don’t trust myself to accurately read between the lines and discern the grades that R1 faculty above mentions. Even if I did, I also don’t trust other philosophers to write between the lines like that in a uniform way. Even if I did I don’t trust myself to not be dazzled by prestige bias. And even if I did *that* I still don’t think I have good reason to trust what’s in the letters over what’s in the cv. An adviser thinking you’ll publish is all well and good. Actually publishing matters drastically more.

Letters are a waste.

[from applicant: letter matters]

Ok, I am going to leave a comment as a former job applicant, not a committee member. I definitely think the letters are very important. I notice that letters are particularly important and well-received if (1) your work is difficult and the letter writer is very good at making your research project accessible and clearly articulating its significance (2) your letter writer is very well connected with the search committee so that the committee trusts the words (3) your letter writer is recent and outside of your graduate program. These are my two cents. Hope it helps.

[Letter of the law]

some obsverations on letters ...
a. one thing I noticed was that when you had two applicants with letters from the same person (or sometimes people) it was very hard to go against their ranking of the candidates. If they said "X ranks among the best", and in Y's letter "Y is a strong candidate", Y was out of the running.
b. most letters were either quite generic or "standard" in content and form and really added nothing that could not be found in other parts of the application. The letter might say she has already published an excellent paper in Nous ... well, I can see that on her c.v.
c. there is a "secret code" used, especially in the USA. I learned, for example, that many people describe a candidate as "a very strong teacher" (or some variant) which means their research is not up to much. These "strong teachers" turn out to be no better teachers (and sometimes worse) than the accomplished researchers. Lesson learned: hire someone who is an accomplished researcher. People at ranked universities were doing this shit.
d. I found as you read more letters about a candidate my opinion of them weakened. That is, I found myself focusing on the weakest remarks about the applicants. So, applicants should try to over impress us with many letters.