The Secret Lives of Search Committees - Part 15: recommendation letters

As a candidate, so many things about the academic job-market seemed hopelessly opaque to me. What were search committees looking for? How did they read and evaluate dossiers? How did they decide who to interview, invite to campus, or hire? I've tried to shed some light on these things so far in this series, at least from my particular standpoint (having served on several search committees and having spoken to people at other institutions who have as well). I hope you are all finding the series helpful.

In any case, one of the things that seemed the most opaque to me as a job-candidate was the role that letters of recommendation play in the process. How are letters of recommendation read? What are search committee members looking for in them? What looks good? What doesn't? As I explain below, my own experience--which once again may be idiosyncratic--is that there is no simple answer to any of these questions. It's no wonder the role that letters play seems so opaque from the candidate's side of the process: it seems hardly less opaque to me on the hiring side (though I do think there may be a few areas of consistency that candidates may find helpful). Allow me to explain.

Recommendation letters & research jobs

Given that I've only served on the hiring side of things at a "teaching-oriented" university (albeit one that appears to be putting increased emphasis on research), and given that the people I've spoken to who have hired elsewhere work at similar universities, there is not much that I can say about the role that letters of recommendation play at research universities. However, as far as I can tell (though please correct me if I'm wrong, if you've hired at a research university), it appears to me that recommendation letters addressing your research can matter quite a bit. If, for instance, Derek Parfit (R.I.P.) or whomever recommends you as the Next Big Thing--and their recommendation is echoed by similar others--I have a hard time imagining that wouldn't make a difference. On the flip side, if one's letters are more tepid, or come from less-well-regarded figures, I have to imagine that might hurt (or at least, not help relative to candidates with the former type of letter).

So far, so intuitive. Is this wrong? Alternatively, are there particular things that stand out for research jobs I'm not thinking of? I happily encourage those who have hired at research departments to chime in - as again, I'd be the first to admit I'm mostly just speculating here. Why speculate at all, then? One reason I wanted to begin with research jobs is I think it may once again be useful to contrast these types of jobs with other types of jobs--"teaching-focused" jobs at liberal arts colleges, universities, and community colleges. For, as I will now explain, my sense is that letters can play quite a different role there.

Recommendation letters & teaching jobs

Let me begin this section with a frank admission: having hired at a teaching institution multiple times, I just haven't found research-related recommendation letters very helpful in evaluating candidates. I will explain why momentarily. However, before I get to that, a few quick points related to this.

Teaching-related letters

While I have gotten the sense that others at institutions like mine may attach more weight to research-related letters than I do, I have also gotten the sense that teaching-related letters play a significantly greater role for teaching jobs than research-related letters do. Why? For a couple of reasons.

First, whereas it is (in my experience) relatively easy to get a handle of what someone is like as a researcher by reading their research statement, writing sample, and other publications, my sense is that it is very hard to get a handle of what someone is actually like as a teacher from their teaching dossier. Sure, you can get the candidate's teaching philosophy - but it can be hard to tell to what extent their teaching methods in the classroom actually reflect that philosophy. For similar reasons, it can be hard to get a handle of how effective the person is as a teacher, as all you have to go on are the candidate's own materials and student evaluations (which yeah...I hope you know the problems with those). Consequently, teaching letters can be really helpful in deciding who to interview. A frank, detailed letter by a third party of what actually you do in the classroom--and frank assessment of what seems to work well and what you could improve at--can, given the rest of the teaching dossier, plausibly give the reader a much better sense of your strengths as a teacher than the teaching portfolio alone. Further, at least in my experience, a frank assessment of areas that could use improvement is not necessarily a bad thing. Indeed, I would much rather read a "balanced letter"--one that highlights a candidate's strengths and weaknesses as a teacher--than an overly effusive one that contains no discussion of areas of potential improvement. The 'balanced' letter is not only likely to strike me as probably more honest (who doesn't have areas they could improve in as a teacher?), but also helpfully informative: I'd like to know where the person thinks the candidate could improve, not necessarily so that I will 'rank them lower' as a candidate but in part to determine whether my own impressions of the candidate are accurate or inaccurate (as one often gets a feel for weaknesses when reading a portfolio anyway!).

That's one reason why teaching letters may play a bigger role than research letters for teaching jobs (though again, my experience may be idiosyncratic): teaching letters may simply be more informative than research letters, giving readers information about teaching that is harder to gain than similar information about research. A second reason, however, is more direct: teaching schools are hiring teachers. Yes, we care about research. Still, teaching comes first. At my school, we are expected to devote something like 45-60% of our time to teaching, 15-30% to research, and 15-30% to service. At some other schools, I get the sense the weight in favor or teaching is much higher, upwards of 75-80%. So, teaching letters are very important for a second reason: hiring an excellent teacher is often our highest hiring priority.

Research-related letters

What about research-related letters, then? In my experience, candidates usually have significantly more of these than they do teaching letters, even for teaching jobs (I had six research letters, and only one teaching one!). What role do these letters play when applying for a job at a teaching school? It may come as a surprise to candidates reading this that my honest answer is: I have no idea. I've gotten the sense from others I've spoken to that research-related letters are certainly read at teaching schools, and can make a difference in some nebulous sense. However, the role that they actually play in decisionmaking remains opaque to me. It remains opaque, in large part, because--having hired three times now--by and large I frankly did not find research-related letters very helpful. Why? For several reasons.

First, my sense is that because research-related letters tend to be overwhelmingly positive, it can be really hard to infer much from them for the sake of ranking candidates. The fact is, most letter-writers (at least in the US, though I have heard things may be different elsewhere) tend to talk up their candidates a great deal. I don't blame them, of course: they are writing in support of the candidate, nearly all of whom are pretty damn good philosophers! The problem is, when just about every letter is overwhelmingly positive, they tend to blur into each other, making it difficult to make heads or tails of which candidate really is the better researcher on the basis of the letter.

Second, my sense is that it isn't necessarily wise to attach greater weight to a letter based on the stature of the author (viz. their status as a Super-Famous Philosopher), as that runs the risk of introducing unhelpful bias into evaluating candidates (viz. is this candidate really better than another just because they wrote a dissertation under Super-Famous Person, who really likes them? Or, is the other candidate better even though they weren't supervised by Dr. Famous?).

Given these two problems, the most natural thing to do then is to read the candidate's materials (i.e. their research statement, writing sample, etc.) and then interpret the recommendation letter on the basis of one's own reading of the person's materials. However, my experience at this point is that one of two things can occur:

1. One can disagree with the letter-writers' evaluations (which in my experience can happen surprisingly often!)
2. One can agree with the letter-writers' evaluations (which in my experience also happens routinely).

The problem then is that if (1) is the case, the letter wasn't that useful (since you disagree with it), and if (2) is the case, the letter isn't that useful either (as you only 'lend weight' to the recommendation to the extent that you agree with it!). Of course, recommendation letters could in principle get you to reconsider or otherwise alter your perception of the candidate's work - buy my own sense, based in part on first-personal experience and partly by talking to others, is that all too many people seem liable to attach more weight to their own estimation of a candidate's work than anything else (particularly given recognition that letter-writers are not exactly disinterested parties).

Anyway, long story short (I know, TL/DR!), while teaching letters clearly play a significant role in hiring at teaching institutions, it is far less clear to me what role research letters play or how much they play it. But again, perhaps my experience is idiosyncratic. I would love to hear from others who have been on the hiring side of things to find out!

Comments

[anonymous]

Hi Marcus,

I both was hired by a research school and have served on multiple search committees at that school--but I do think things are different even among research schools, so I don't think I speak for everyone here.

--Yes, letters are (typically) uniformly extremely positive. Still, every time we've done a search, there are (depending on how broad the search is) 1-5ish candidates whose letters are overwhelmingly stronger than everyone else's. I'm not sure how this happens; but it's very noticeable (this probably also helps explain--presumably, in conjunction with excellent writing samples--why certain people who might not seem like their c.v.s are the world's greatest get lots of fancy offers).

In my department, we take letters very seriously, but they would never trump a writing sample that we didn't like. Sometimes hints of negativity can hurt a candidate, but if we otherwise like the candidate we will attempt to dig deeper, and if it is only in one letter we will sometimes discount it given that it might just be that that candidate didn't have access to having her letters vetted, someone just had a personal issue with her, etc.

I know people say letters are useless because they are so over the top. Part of me thinks that is true, but then, whenever I am on a search committee, they end up being super useful to me. They often are better at communicating what is interesting or important or novel about the candidate's research than the candidate's own materials are. They help paint a fuller picture of a candidate as a philosopher than the candidate could on her own. And despite the fact that they are nearly uniformly over-the-top positive, there are ways of differentiating between them.

I think one of the most important things for research schools is having an outside letter writer who thinks extremely highly of you, especially if that person has their own PhD students who are competing with you for jobs. There is very little pressure on such a person to write you an over the top letter. If they do, I take that as strong evidence. If they write you a stronger letter than they do their own students (rare but happens), I take that as very strong evidence.

Also, I think that yes, there is all sorts of room for bias introduction here. But from the perspective of the hiring department, some of that bias is reasonable (if we know and trust a letter writer, we have more solid evidence about the candidate. If we can contact a letter writer, we have even more solid evidence...).

[anonymous]

p.s. My best guess is that only a handful of my colleagues ever look at teaching letters. I look at them because I think it matters whether we have good teachers in our department, and I also think there is often evidence of other important things (e.g. will this person gladly do service?) in them. But I don't remember a single person ever discussing the content of any of them in a hiring meeting. (Also, some people don't submit them to us.)

[Amanda]

Anonymous, from what I understand, it is not just that research schools do not take teaching letters seriously, but also that they do not take teaching seriously, correct? Please correct me if I'm wrong. I got the impression almost no time went to discussing a potential hire's teaching abilities. (There might be an exception if the program really wants to start say, a certificate in Law, and so needs someone to teach a lot of philosophy of Law) I do think that *some* individual search committee members, like yourself, care about teaching and that might influence who they recommend for an interview. But it does not follow that teaching is ever discussed as a group.

Marcus, what about letters that discuss both teaching and research? I think 3 out of my 8 letters did that. (And no I did not use all 8 at each place...at least not most of the time. I did at a few places.)

[Duonymous]

Regarding outside letters and research schools: If you're in a small field with a few prolific letter-writers (niche areas of history come to mind), don't be surprised if almost all interviewed candidates have letters from the same 2-3 faculty. (I was explicitly told this on a fly-out, and I've also heard it from others.)

Two take-aways from this: First, you often do need that outside letter to get in the door. Second, once you get in the door, the school may have a letter which may directly compare you to several of the other candidates, so it's all the more important to be professional with your outside letter-writer, do your best to give them a sense of your research, etc.

[anonymous]

Amanda, I think it varies. We've discussed teaching before, but rarely. I do think that most of us at least look at a teaching statement and glance through teaching evaluations. There has been one time that one of the candidates we were considering had very low teaching evaluation scores and his teaching statement was weird (in a bad way), and we decided not to interview him basically for this reason. But I do think at least at my university, the conventional research-school wisdom (teaching statements and letters can only hurt and not help you, teaching scores and evaluations don't really matter unless they hurt you) is mostly true. Evidence of bad teaching can hurt you, but evidence of good teaching does basically nothing.

But I have friends who have been interviewed at R1s who got a lot of teaching questions/discussion of their teaching, which seems to indicate that those departments actively care about teaching when hiring.

[Amanda]

Thanks anonymous. Yeah, I wonder whether the questions about teaching I got at research schools were just for show. I would think if they cared much they would have a teaching demo. But it is heartening to know terrible teaching can take you out of the running.

[Marcus Arvan]

Hi Anonymous: Thanks for sharing - that's all very helpful!

I haven't had the experience of 1-5 candidates having far better letters than everyone else. I wonder whether this might be unique to R1 jobs, where it seems likely that each season there will be a few Rock-Star Researchers who stand out head and shoulders above everyone else (and who might therefore only apply to R1 jobs).

Anyway, I wonder if this is what is going on. My own experience hiring at a very different type of institution has been that very few research letters stand out--in which case as a search committee member one really has to make up one's own mind reading candidates' other materials. But again, this is just my experience, and could well be idiosyncratic.

[SM]

Just a practical question: Many places these days only allow you to upload 3-4 letters. So, Marcus, are you saying that, for instance, if you apply to a teaching school, and can only upload 3 letters, that 2 of those should be teaching letters? At my program, we're instructed that we only need 1 teaching letter. Would you recommend getting two people to write about our teaching, then?

[Amanda]

I am confused by this whole teaching/research letter thing? Can't a single letter address both research and teaching?

[Marcus Arvan]

Hi SM: Good questions. I'm not entirely sure, but I suspect one teaching letter will suffice - as it only takes one really well-written letter to give committee members a clear picture of what you are actually like as a teacher. I only used one teaching letter and I did just fine on the market - and as Amanda's comment notes, other letter writers can comment on your teaching as well.

Amanda: Yes, of course, a single letter can address teaching and research. But a lot of letters that do both focus much more on research than teaching. So, I think it's important to have one very thoughtful letter written only on teaching - preferably by a faculty member who cares about teaching and has sat in on some of one's classes.