Neurofunctional

Our books






Become a Fan

« Using other offers to renegotiate one's position | Main | Twitter thread on negotiating offers »

02/21/2020

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Overseas TT

I don't care about letters, at least about candidates in my own AOS, because (to put it bluntly) I think I just know my field well enough to trust my own judgment about candidates without having to know what other people think about them.

The one and only reason I take a look at letters is to see if there's any red flag about a candidate as a colleague - i.e. whether someone is difficult to work with or makes for a toxic work environment. This is very rare but I've seen/heard examples of it. Other than that, I couldn't care less about the letters, no matter who they are coming from.

anonymous grumpy philosopher

I have mixed feelings about letters, but when it comes to job candidates, very often no one on a hiring committee is a complete specialist in the area a candidate works in, and I think that while letters are often inflated and sometimes unreliable, they can provide helpful information from experts in a candidate's specialty.

Think about schools that use external letters for tenure; there, I take it the idea is (a) that people with significant expertise in a tenure candidate's subfield is going to be in a better position to evaluate that candidate's work than their colleagues may be and (b) by agreeing to write such letter, external writers agree to read the candidate's work carefully--not just one paper, but most or all of their work--and form a complete judgment about it--which the candidate's colleagues are (in most cases) unlikely to do.

I think both of these things translate to the hiring context. Someone's advisor, while obviously biased in certain ways (usually) in favor of the candidate, also has spent WAY more time reading the candidate's work, looking at drafts, having a complete picture of the candidate as a philosopher than the candidate's writing sample, cv, etc. can portray. So they are in a (b)-like position in comparison to the hiring department. Also, someone's advisor is way more likely to be much more of an expert in the thing the candidate is working on than the hiring committee is.

While I think there are many reasons to discount letters, I think ignoring these positive reasons in favor of taking them seriously amounts to a kind of intellectual arrogance. (You can of course think that these reasons are outweighed by the reasons against letters. But I think pretending they are not reasons is problematic.)

Marcus Arvan

grumpy: thanks for weighing in. Just to be clear: I don’t think we should ignore that there are some positive reasons in favor of taking letters seriously. I just think that the balance of evidence is that the negative reasons generally vastly outweigh the positive ones. Indeed, if epistemic humility is what you are after, then I think the best positive and most epistemically humble measure of someone’s research promise is their actual publication record. Why? If a person has published articles in journals X, Y, and Z, then that person has demonstrated that their work is good to multiple journal reviewers and editors who (due to anonymized review) are likely to be (A) specialists, who also (B) have no direct connection to the author and/or knowledge of who they are.

From an epistemic point-of-view, I’m inclined to think we’re much better off deferring to the judgments of multiple specialists in a peer review process designed to evaluate work on its merits over the judgments of 3 letter writers who have direct connections to the author—especially when we have independent scientific evidence that (C) letters of recommendations are poor predictors of future success, (D) tend to reflect particular kinds of biases, and (E) past publishing record appears to be the best predictor of future academic success. For studies on (E), see:

https://academic.oup.com/bioscience/article/63/10/817/238191

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6174868/

Jonathan Ichikawa

I think most letters are pretty irrelevant, but not all of them. I try not to pay any attention to how sincere someone sounds when they say the student is great, or to mere evaluations ("I thought her term paper was outstanding") as opposed to specific illustrations ("her paper brought out such-and-such point, which was a really useful innovation because..."). But letters can be useful when they contain actual information not found or highlighted elsewhere in the file.

For instance, a letter might be where I get the narrative to contextualise changes in the students' grades over their career. (This can come in personal statements too but it sometimes better from letter-writers who also know the situation firsthand.) If they talk about how the student took their course in their first year and struggled for some particular reason, then had a real moment where everything turned around for them, this will bear on how I understand low early grades in the transcript.

Letters are also informative when they tell me more specific things about CV items. What kinds of things did they do when they were your RA? What kind of material was covered in that logic class? This bears on what I should expect them to be familiar with already when they show up in my program.

elisa freschi

@Marcus, coming from continental Europe (where letters are either not used or less relevant and are anyway not (that) inflated), I am curious about the evidence you mention against letters.

I checked one of the articles you linked to (Aamodt et al 1993), which sounds quite interesting and sees letters as problematic especially when the author does not know the applicant well enough. Their conclusion is at the end positive (if developing further Peres and Garcia's method, etc.).

The other article (https://psycnet.apa.org/buy/1998-10661-006) is behind a paywall. My first question would be whether it is based on the US model of letters. Do you know? Thanks!

Marcus Arvan

Hey Elisa: thanks for chiming in, and good questions! The problem here, as I see it, is this. Very few people know about let alone use the Peres and Garcia method. A large plurality of studies have found that when letters are simply read—as is presumably done by philosophy search committees—they have little predictive validity. So yes, if people in philosophy started using the Peres and Garcia method, then letters could be helpful—and if they do start to use the method, I would be pretty happy! The problem, though, is that few (of any) people presumably use the method now, and few will presumably use it in the future (since by all accounts philosophers prefer to simply read letters to draw their own conclusions). This is a more general issue in hiring (which I learned from my spouse, who specializes in the area). Researchers like Peres and Garcia came up with their method nearly 60 years ago (!), yet so many decades later very few people use the method. For example, had you ever heard of it before? Do you know anyone who uses it? The problem then is that unless academic hiring committees start to use it (and they probably won't), then letters will remain the poor predictors for all intents and purposes.

I can’t re-access the second study while at home this weekend (I only have access to paywalled articles at work), but my recollection is that it was a meta-analysis of 80-something studies examining the predictive validity of letters in general, and the paper reports that they have among the lowest predictive value of any commonly used measure. I don’t recall whether the meta-analysis focused on the “US model” or not, but I don’t think it did!

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)

Job-market reporting thread

Current Job-Market Discussion Thread

Cocoon Job-Market Mentoring Program

Categories