Our books






Become a Fan

« How often do interviews make a difference? | Main | Must a writing sample fit the job? »

12/11/2020

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Evan

The issues with letters of recommendation arise because we still haven’t adequately addressed the nature and aims of letters of recommendation in the first place.

The researches that showed there is extremely little correlation between letters of recommendation and job performance showed that because the researchers presupposed that in the first place. In other words, they assumed that that’s what letters of recommendation supposed to do: give a glimpse of a candidate’s skills, knowledge, and competence.

If that’s how you understand letters of recommendation, then it’s obvious that the evidence will suggest that it does little to show you a candidate’s future job performance.

However, the researchers failed to consider that letters of recommendations are broad. They can include various descriptions of a candidate. One of them being the moral description of a person; description of a person’s character and behavior. Moral character has little to do with job performance per se. It’s more about how this person behaves with and among others on the job. It’s a matter of integrity and whether their character aligns with the ethos of the organization or institution.

Is this person friendly? Is this person punctual? Is this person responsive? Is this person responsible? Is this person motivated? Is this person detailed oriented? Is this person helpful? Is this person more passionate about or interested in research or teaching or both? If so, how so? Etc.

I’ve always thought letters of recommendation is about describing a person’s personality, character, and patterns of behavior on the job. Thus, I think if we understand letters of recommendation as some form of evidences as to whether this person will be a good or compatible colleague or teacher, then perhaps letters of recommendation can still be of value in that realm. I just don’t think they should be used to determine or predict present and future job performance, knowledge, or competence per se.

I’ve read my high school teacher’s letter of recommendation of me after I got accepted into college since we can only view it once we got admitted. Not once did she mentioned whether I was good at the subject or competent at something. She focused on my character and how I behaved in class and with other students.

The researches done on letters of recommendations make sense if we just focus on job performance and overlook character and intra-office/work environment. The researches made sense because one can be very competent and skillful and yet still treat one’s colleagues, patients, clients, or students poorly. Conversely, one can still be very friendly, but not as skillful or competent compared to others.

Transcripts and publications will not be able to tell you if this person will be a good colleague or teacher to work with. So many people here complained that having a toxic colleague is bad and they want to avoid it as much as they can. But if we’re being honest, we have few evidences to go by that will tell us whether or not somebody will be a good colleague or teacher to work with. Letters of recommendation are one of them.

op

Op to the commenter

I never said the letters should be ignored. I said they should not be weighed any more heavily, which clearly looks to be the practice. Another thought behind the scenes of my comment was something like, 'well, if you're not going to trust my supervisor's letter, but you are going to trust the external letter, and, indeed, weigh it more heavily, why don't we just only send you external letters?'

By the way, I'm a graduate student at a middle-ranked Leiter, but I'm not sure what that has to do with anything. If your point is that external folks will look at my program and think, 'nah, rather write for my ole Harvard chums students at Princeton,' then you're just proving my point that this market phenomenon is just a reflection of the incestual relationships that exist among top programs and their respective students. Don't get me wrong, I'll play ball. I'm just trying to learn more about the rules of the game.

RecentTT

I completely agree with Evan that letters of recommendation are about soft skills. And I believe that soft skills matter in philosophy too. We have to teach after all and our research takes place in a scholarly community. Marcus once used the NFL draft as analogy for our hiring processes. Using this analogy, I would characterize recommendation letters as our background checks (by the way an important, more or less hidden part of the NFL draft where the difference between Manning and Leaf was obvious...). I was part of hiring committees and know that colleagues use recommendation letters in exactly this way, also by pointing to potential issues with a candidate (again, think about Leaf...).

I believe that there are several reasons why external letters are more valuable (most of them are implicitly or explicitly acknowledged by the OP): 1) There is less self-interest (it is important that your students get a job!) and personal loyalty (often distorts the judgement) involved. 2) It shows that the candidate is well connected outside her department. It is a real achievement to get someone outside interested in your work. Maybe you were admitted to a conference, someone might have been impressed by your talk, and you had a great conversation afterwards etc. Or you organized a great workshop etc. And yes, some have it easier to get external letters, since their superstar supervisors from wealthy departments are well connected. But where do these candidates do not have it easier? They have more influential internal letters too. Sometimes external letters might be capable of counterbalancing this inequality a little. I know that they did in my case. I had two external letters at the end of my job circle and they demonstrated that I am well connected to the US context--something that was very important for someone who comes from a non-US place...

RecentTT

I also have a question for Marcus: Are the studies you are referring to about philosophy in particular? If not, which field are they examining? I ask, since I would be very cautious to apply results from a STEM field to philosophy. There might be some corners of philosophy that are similar to STEM fields, but most of philosophy is simply not. My point is: we would need scientific evidence about which factors predict future success in philosophy...

Marcus Arvan

RecentTT: philosophy in particular has not been studied to my knowledge. But these results have been replicated for over 40 years across many different fields, and they always come out the same way. People in every field think things like letters of recommendation and interviews matter, and that these things provide some kind of crucial insight into 'expertise' or 'soft skills.' Then it turns out that, no, the best predictors of future academic and workplace success are objective measures of past performance. See e.g., https://psycnet.apa.org/record/1997-02834-005

Indeed, studies have been carried out in academic contexts, and what do you know? The find the same thing: tangible accomplishments are the best predictor of future success. https://www.nature.com/news/computer-model-predicts-academic-success-1.15337

The reasons for this are actually pretty straightforward. They have to do with human biases. Human beings have a litany of cognitive biases that do little more than introduce irrelevant noise into evaluating evidence. See e.g. https://en.wikipedia.org/wiki/List_of_cognitive_biases

Things like letters of recommendation and interviews appear to do little more than play into these biases in all kinds of pernicious ways. For example, judgments of interview performance have been demonstrated to be influenced by attractiveness, weight, height, gender, race, speech style, and voice-timbre--and interviewers have been demonstrated to be completely unreliable when it comes to determining whether an interview is honest or lying. Similarly, letters of recommendation have been shown to be influenced by things like gender, etc.--not to mention selection biases, etc. The problem then is that none of us think these biases are actually at work in us when they are.

People in my spouse's field have not only researched these things over many decades, finding the same things over and over again (again, across a wide variety of fields). They also work in applied settings, i.e. revamping the hiring practices of major companies, government agencies, etc. And when they implement the kinds of changes supported by the science, what do you know? Hiring outcomes improve (i.e. companies hire workers that succeed on the job more reliably).

Nobody wants to hear this, of course. We all want to believe that we have some special expertise in detecting promise, etc. The problem is, the science just doesn't support it. Is it possible that philosophy is an outlier: that it's one of the rare fields in which letters of recommendation and interviews are good predictors? Sure, anything is possible. But the overall balance of empirical evidence on this stuff, as I understand it from my spouse, is very clear: if you want to hire the best people, hire on the basis of accomplishments.

RecentTT

Thank you, Marcus, for the evidence (and sorry for misusing this thread OP). I, nevertheless, do not understand your crusade against interviews and recommendation letters. First, I do not like your suggestion that other people do not hire on the basis of accomplishment. I look at the objective accomplishments first. Yet my experience is that you always have several candidates (mostly even more than the ones that get interviewed) who have a similar level of accomplishment (esp. if you weigh it against their academic age). Hence, I need additional criteria. Here the job talk, interview performance, and recommendation letters come into play. The other option would be to draw lots among the candidates who are equally qualified (an option I would seriously entertain). Finally, I see some issues with your focus on "accomplishments": biases are also involved when it comes to what we count as "accomplishment" and how we weight them against each other (e.g. which journal is better etc.). These biases already start in primary school, by the way. There is so much sociological research that demonstrate how disadvantaged minorities (including working class) are when it comes to how their performances are evaluated in school. I do not know how your computational approach is able to track the social structures that disadvantage certain people from the very beginning. For me, your approach sounds like introducing a veil of ignorance and I believe there are legit objections against such an approach...Another critical line of thought would be to point at Johnny Manziel (Heisman Trophy Winner) and careers like his...

Marcus Arvan

Hey Recent TT: no worries, I'm happy to discuss this. The reasons that I argue so strongly on these issues are two-fold. First, I'm married to someone who works in the area, and who has convinced me that this stuff (hiring on the basis of good science) is important. Second, as I've noted in the past (and my spouse has impressed upon me), things like recommendations and interviews tend to disproportionately harm the already marginalized on the performance-irrelevant bases (e.g. race, gender, disability, attractiveness, etc.). Because as a philosopher (and person) I care both about truth and justice, it seems important to me to get the word out that traditional ways of screening and hiring people are problematic.

To be clear: I'm *not* saying that people don't hire on the basis of accomplishments. Clearly, they do. We read candidate files, evaluate CV's, writing samples, etc. The problem, from my perspective, is that we then muddy the waters with selection methods (interviews and letters) that have little predictive power and introduce all kinds of insidious biases. That's all that I have issue with!

Now, I think you're absolutely right here on two things. First, the science actually supports two of the additional criteria you say you need: things like job talks and teaching demos. The science indicates that these things *are* valuable, because they closely model on-the-job tasks (whereas interviews don't). So, I'm all for these kinds of screening tools--and notice: some universities (such as Princeton) have eliminated first-round interviews and use these methods instead. Princeton (or so I've heard) goes straight to on campus visits, which is exactly right I think. Second, you're right that measures of tangible accomplishments can also be biased. This is a big area of empirical research as well. However, there are ways for counteracting these kinds of biases--such as examining the ways in which particular measures of tangible accomplishments are biased (on race, gender, etc.), and using selection criteria that compensate for such biases.

In terms of Johnny Manziel, I don't think he's a very good case here. While his on the field accomplishments in college were spectacular, he had numerous off-field incidents while in college that wise NFL teams took to be red flags--behavior that he ended up repeating in his time in the NFL and ended up tanking his career. Here's a short list of his bad behavior in college (including criminal behavior): https://www.si.com/nfl/2016/01/26/johnny-manziel-career-controversy-timeline#:~:text=While%20enrolled%20at%20Texas%20A%26M,possessing%20a%20false%20identification%20card.

Notice that this too is a case of past performance predicting future performance. This, among other things, is why government agencies (such as the NSA, FBI, etc.) deeply probe applicants' personal backgrounds and why employers in general do background checks prior to hiring people. The science of hiring by all means supports this, and the problem with Manziel is that the Browns looked past his off-the-field shenanigans when drafting him.

Evan

It’s easier to hire people based on expertise and accomplishments because things are made transparent via transcripts, test scores, certifications, degrees, experiences, etc.

I do agree that if you want to hire the “best” people, then look at their past accomplishments. Certain long gaps between different years should be noticed as well. But again, “best” is very vague.

Let me provide a concrete example: nursing. I have a lot of friends who are RNs. They constantly tell me that some of their co-workers are problematic such as being very lazy. And lazy nurses can kill patients. Some nurses allow their politics to affect their medical judgements because some of them think that Covid-19 is a hoax devised by the Democratic Party. Some don’t even believe in basic germ theory. We’re led to ask: how is it possible for these people to work given their flawed character and behaviors? I suppose their general expertise and doing the bare minimum are good enough for these hospitals or clinics.

I look for both competence and good intentions. The institution should exhibit both from the front desk to the back of the building. I took my father to see an ophthalmologist. The technician and front desk were friendly and competent. But the doctor himself was impatient, rushy, and condescending. He was competent and knew what he was doing of course, but I still wouldn’t go back again.

I don’t think most graduate students are toxic or will be bad colleagues once they secure a job. I think if they are given the position they truly desire, they may do very well. They can be problematic when they’re forced to do a job they hate such as teaching when they just want to be left alone and do research all day. People who hate their jobs are usually the ones who tend to do the bare minimum or become bad co-workers. You can tell if somebody loves their job or at least takes it seriously.

I’m at a point where when it comes to the character of a person in the university or school setting, it should not be taken too personally. Some bad teachers are just forced by this system to work at a job that’s only available for them. That’s why I encourage independent learning and autonomous thinking for students to compensate for ineffective teachers. There will always be brilliant, but lazy or problematic workers out there. Bad teachers don’t bother me anymore because I am at the point of my cognitive ability to be able to teach and learn by myself. This is from a student perspective.

Evan

My conclusion to whether or not letters of recommendation are useful or valuable really depends on the function and aim of your university, department, and professional work environment.

If you’re looking for a candidate who can pump out lots of papers every year, if your university is not teaching focused, and if you don’t really need to rely on or cooperate with your colleagues much, then you can pretty much disregard letters of recommendation. In such an environment where there is little to no interaction between co-workers and teachers and students, then one could care less if they’re being responsive, friendly, helpful, etc. so long as they pump out lots of research as expected of them.

For example, a chemist working at a chemical testing facility really just stays in the lab all day testing certain drugs by themselves with little interaction with co-workers or other people would not really require a letter of recommendation I would think. A relevant degree and work experience would be suffice. Just stay in the lab all day and tell the police department know whether this substance sample is cocaine, meth, sugar, salt, etc.

But if you wanna work at Starbucks, then you probably need one since they are notorious at desiring people who are or can be very friendly and cheerful. I witnessed this first hand when I overheard one of the co-workers telling the cashier to be more engaging with customers. I rarely ever met a grumpy Starbucks employee.

Jobs that require lots of mutual interaction and dependency among workers and others would probably require letters of recommendation to give some evidence of their behavior with other people. And jobs that are very autonomous and atomistic probably wouldn’t.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)

Job-market reporting thread

Current Job-Market Discussion Thread

Philosophers in Industry Directory

Cocoon Job-Market Mentoring Program

Categories