More news on the interviews-are-worse-than-useless front

I've written before on the topic of interviews: namely, on how decades of empirical research shows that:

1. Interviewers' judgments of candidates are worse predictors of job success than purely algorithmic procedures using quantitative data alone, most likely because
2. Judgments of interview performance tend to track characteristics of job candidates (such as attractiveness, weight, height, gender, race, speech style, and personality traits) that are either entirely unrelated or poorly related to successful job performance, and
3. Judgments of interview performance fail to reliably track interview behaviors (such as lying) that predictably confound interviewer judgments of candidate quality.
4. And yes, videoconference interviews have substantial bias-effects of their own.

Now we have a fun new result: "vocal fry" hurts women job applicants . All this evidence...and yet the world simply continues to turn...

Comments

[Trevor Hedberg]

Hi, Marcus. Consider this a continuation of our discussion from your earlier post. I'll frame my comment around two questions.

First, what do each of these studies consider "interviews"? There are at least two types in philosophy: shorter interviews (usually conducted over Skype or in-person at the APA Eastern) and longer on-campus interviews that span multiple days. It's important to know whether these biases significantly distort judgments even in the lengthy on-campus interviews because one possible alternative to the status quo would be to eliminate Skype interviews and exclusively conduct campus visits for the finalists (though this might mean that the number of finalists a department selects is higher than the usual 2-3). Since the on-campus interviews are longer and involve a greater diversity of interactions (e.g., dinner with faculty, lunch with students, a teaching demonstration or research presentation), it's not obvious that data on shorter interviews would be replicated in on-campus interviews. I assume the biases would always be present, but perhaps the greater amount of time and engagement enables the interviewers to "correct" (at least to some extent) their distorted judgments.

Second, given that you are decidedly against job interviews, are you advocating an algorithmic procedure for hiring applicants? If so, how could such an algorithm be formulated? This would require answering a lot of tough questions. How much weight is to be given to publications in journal X compared to journal Y? How much weight should be given to a person with a PhD from a top-20 Leiter school compared to someone from an unranked institution? The big-picture worry is that the particular weights given to these factors might be arbitrary, and if that's true, it's not clear that the algorithmic procedure is an improvement over the interview procedure: the arbitrariness in the algorithmic procedure would have to be shown to be less severe or more benign than the arbitrariness in the interview procedure.

Regarding this worry, I tried to find information about Princeton's hiring procedure (since you mentioned in our previous exchange about this topic that Princeton doesn't conduct job interviews) to see what broad criteria they might have formulated for such a decision-procedure. But what I found suggested that they do conduct interviews for academic positions -- http://www.princeton.edu/hr/employment/managers/process/hireprocess/ (see step 7) -- unless they've changed their policy recently and have not updated their human resources website.

There is also a worry about genuine incomparability among job applicants. When I try to compare some CVs, it strikes me that they are genuinely incomparable with one another -- that is, neither CV is better than the other and they are not equal; they're just too different to admit of a sensible comparison. I suspect that the same could be said about candidates' application materials as a whole. So here's the worry: suppose your algorithm generates 5-6 candidates that are better than all others but cannot meaningfully differentiate between the overall merits of these 5-6 people. If you aren't going to conduct any interviews whatsoever, then what do you do? Pick at random?

[Marcus Arvan]

Hi Trevor: thanks for your comment, and great questions!

The way I understand the empirical data, actual job-related performances (e.g. teaching demos) are *substantially* better than interviews--and so, yes, I would advocate foregoing APA and Skype interviews in favor of going straight to on-campus interviews. Interestingly--and, I would say, encouragingly--this is a direction that an increasing number of programs seem to be going!

In terms of your second question, "are you advocating an algorithmic procedure for hiring applicants?", the answer is: yes, I am--the empirical research *unequivocally* shows that this is a more reliable process for selecting successful people than a non-algorithmic process (note: this is actually not controversial in Industrial Organizational psychology. It is widely accepted).

You're certainly right: developing an algorithm is difficult. Interestingly, however, the data shows that however one plausibly develops them, they *tend* to do better than human judgers. This is because, although any algorithm is (as you note) subject to biases of its own, algorithms still *minimize* bias--by, for instance, eliminating irrelevant factors such as weight, height, attractiveness, speaking tone, gender, etc. Once you think about this, it's actually quite intuitive. *No* measure can entirely eliminate bias. But what algorithms can do is eliminate a great deal of bias, and determine decisions on things (e.g. publication record, etc.) that are actually closely tied to job performance--whereas non-algorithmic measures (i.e. interview judgments) introduce a variety of confounding biases. Thus, as problematic as algorithms may be, they *still* tend to be better for these reasons than human judgers.

In terms of Princeton's hiring practices, that might be their official policy--but the rumor going around (and it may well be false) is that they don't follow it. It wouldn't be the first time someone didn't follow their own policies, but again, the rumors may be erroneous.

In terms of your final question, "suppose your algorithm generates 5-6 candidates that are better than all others but cannot meaningfully differentiate between the overall merits of these 5-6 people. If you aren't going to conduct any interviews whatsoever, then what do you do? Pick at random?", the answer is: I would pick at random, or else pick the person I liked and recognize that my preference is probably arbitrary. And I don't think either conclusion is unwarranted. In my experience, almost everyone on a search committee will tell you the same thing: once you arrive at the final 5-6 people, they're ALL great picks. You might as well pick straws at that point.

[Trevor Hedberg]

Ah, okay. Now that I better understand your position, I don't think we disagree as much as the previous exchanges would suggest. My suspicion is that the main value of interactions with candidates would emerge during full-fledged campus visits, so if your position is compatible with keeping those as part of the process, I'm not sure I really have any objection. (When I reread the post from January, it occurred to me that we might have been using the term "interview" differently -- you using it to refer only to the shorter one-on-one interviews and me using it more broadly to include both those and campus visits.) I know a few departments that went straight to on-campus interviews in the recent past, and I can see how that switch could be beneficial. Still, I'd hope that a careful, lengthy discussion about how the different aspects of an applicant's CV and other materials are to be weighted would precede the shift away from those intermediary interviews: you want to be as certain as possible that the method you're adopting is better than the method you're abandoning.