How do search committees sift through apps?

In our job market discussion thread, a reader asks:

Does anyone know how search committees typically sift through applications? Does the search committee look at applications before the deadline? Does every member of the search committee look at every application, or do they divide up applications before making a first cut?

These are great questions, and I expect many job candidates would be interested in hearing answers! Any search committee members willing to weigh in? In addition to answering the OP's questions, it might be great to hear a little as well from search committee members on how they actually read and evaluate apps. For example, how long do you spend, on average, on a candidate's file? Which parts of the file do you look at first, and what do you focus on? What matters to you most when deciding who to interview or place in the 'don't interview' pile? Etc.

Comments

[just me]

For some of the searches I was involved in each member of our small department looked at all the applications. Everyone did it their own way. And we first sorted applicants into three piles: viable; possibly viable but questionable; and not viable (usually because they did not work in the advertised area or were ABD). The things I tended to focus on were: cover letter and c.v. They say so much. If there were questions about what is there, then I would look at other materials, hoping for clarification. Collectively, we reviewed all those who were deemed viable by all, and those who might have been classed as possibly viable by some, if someone made a case on their behalf. Then we looked at the files in my detail and more completely.
Things like letters were most useful when applications were coming from the same institution - then the same letter writer provided a means for determining who was the best of the files from their school.

[ABD applicant]

@just me: when you say that ABD candidates automatically went in the "not viable" pile, does that refer only to applicants who would still be ABD at the job's start date or to ALL ABD applicants?

[R2 land grant]

When I sat on a TT search committee, I think we had about 200 applications (for an Ethics job). There were three of us on the committee and we all looked at all of the applications. We weren't supposed to look at them before the deadline, but I can't confidently say we all followed that.

In terms of how I proceeded when looking, I did a first cut by looking at just CVs and cover letters. If people didn't plausibly do ethics, or if they had their PhD in hand but didn't have any publications, they were out. Then I looked at research and teaching statements. If I thought their research actually sounded interesting (I'm not really interested in technical problems that don't have very wide implications for other questions) and they actually sounded thoughtful about teaching (I cared more about them sounding thoughtful and actually demonstrating originality in their teaching, rather than how much solo experience they had), they made the next cut. At that point I read writing samples. I looked at letters of recommendations and teaching evaluations last, and only to look for red flags - I think letters of recommendation tend to be way over-inflated, and teaching evaluations tend to be biased in objectionable ways.

Each committee member made a list of (I think?) their top 20, and then each of us looked carefully at all of those folks. Then we each ranked our favorites, and had a meeting where we decided which 12 to do zoom interviews with.

I have no idea how representative my process was - I'm actually really curious to see what others say!

[SLAC Associate (and frequent search member)]

I'd be astonished if there's anything generalizeable that can be said here. Every search I've been on has been run differently, and even within a single search different members of the search committee will often focus on different parts of the application as critical. If anything is a common practice I would think it'd be that every single application gets looked at by multiple people in the first round (though maybe only by a proper subset of the committee), but I suspect that's not even always true. Some committees will start with a close read of every application in order to go straight to an interview list (say, by having every member of the committee pick their favorite five candidates), while other committees will go through a longer process that involves first narrowing the pool by only looking at CVs, then narrowing further on the basis of letters, etc.

For my own part, I think it's important to pass eyes over every component of the application -- if we asked for the document and the candidate spent the time preparing it, the least I can do is take a look at it (though I know lots of people disagree with that). I always start with the CV, and there are some candidates where 5 seconds of looking at the CV makes clear they're not a viable candidate for the search (e.g., they are not even in the neighborhood of having the right AOS). Even for those candidates I'll at least glance at their other documents to make sure there's nothing to counter that initial impression. But it's not uncommon to spend 20 minutes or more pouring over a file to make a determination. My typical reading order at our SLAC is to start with the CV, then the cover letter, then the teaching portfolio, and finally the letters of recommendation and the writing sample (sometimes going back and forth between those last two to let the letter writers help me understand the context and value of the research).

[Tim]

At my institution, the chair of the committee distributes all of the applications after the deadline. Each member of the committee looks over each application. A list of created of the top candidates for interviews using basic scoring rules.

There are no agreed upon methods for evaluating candidates. Like most, I first look at CV and cover letter as a way of eliminating candidates. After that, its mostly guessing.

[Bill Vanderburgh]

I've chaired three searches in my current department and I've served on other search committees at a previous institution.

While applications are coming in, the department chair and I split the job of checking for minimum qualifications (all parts of the app submitted, PhD or ABD in Philosophy). After the deadline, we forward the list of minimally qualified applicants to the Provost's office, and they "certify" the pool. (This is mostly a check that the demographics of the applicant pool are suitably diverse.)

So, about a week after the application deadline, search committee members can start reviewing files. Given that we are all teaching, grading, etc., we give ourselves ten days or so to do this review.

In the first round, each file is reviewed and rated independently by two different people. The point here is to decide who to invite for a preliminary Zoom interview, so we aim to sort files into "yes, maybe, no". We all do it slightly differently but mostly rely on the c.v., cover letter and diversity statement at this stage, looking to the writing sample and reference letters only in edge cases or to investigate something that was mentioned in the other parts of the file. Most of the "no" files can be determined in just a few minutes--lack of requested research and teaching areas is the most common reason. Some "yes" files are similarly very quickly obvious. The "maybe" ones require more time and will have us read more of the file, more deeply. (As someone else in this thread mentioned, letters of reference are generally so positive that they are pretty unhelpful in distinguishing candidates.) Things are so competitive these days that it is unlikely someone would get through to the Zoom interviews without some publications, even though we are a teaching-focused university.

Our goal is to Zoom with 10-20% of the pool. Everyone who gets two "yes" ratings is invited. The committee discusses "yes-maybe" and "yes-no" cases, often going back to look up details in the application during the discussion. Sometimes this process uncovers a reason to add the person to the Zoom list; just as often, it reveals a reason not to do so.

Once we have our list, we forward it to the Provost's office, and then wait for approval. They sometimes complain that the list is not diverse enough (but they also don't share the demographic data with us so it is hard to satisfy them!).

Once we have approval to proceed (roughly 3-4 weeks after the application deadline), we can begin to arrange Zoom interviews. Before those interviews, we read the files thoroughly. Questions in the Zoom interviews are partly universal, partly tailored to the details of the file. We ask about teaching, research, and service (those are the things we are ultimately going to tenure someone on, so we need to feel confident that they could succeed in those areas). We are equally interested here in how well the candidates would fulfill our departmental teaching needs and preferences. Because they passed first review we expect they will be able to meet those needs and preferences, so here we begin to separate candidates on their strengths in the areas, their overlap with our other current areas, and in terms of qualitative evaluations of their teaching. Again, things are so competitive these days that minimally meeting requirements is not enough to stand out.

Plan for seven days to arrange the Zoom interviews, and up to ten days to conduct them (hopefully shorter, but schedules for six faculty and a candidate don't always align easily, and we have to find that magic time for 10-15 candidates).

Then we have another meeting to select campus interviewees, another round of Provost/diversity approvals, and then finally we can move to arrange campus interviews. Making contact to start arranging those interviews might happen, say, 8-10 weeks after the application deadline. (Holiday breaks make this take longer than it might overwise.)

[EuroProf]

Here's how we do it (large university in continental Europe):

(1) An assistant screens all the apps as they come in and puts together a list of all the candidates and the key facts about them, like: current position, where they got their PhD, what it is about, what degrees they have, relevant experiences etc. At this stage, it's mostly your CV and research experience which counts.

(2) Then, once the deadline is over, all the committee members get this list and all the applications. We then study the list and take a closer look at all the candidates who seem to be relatively interesting. Not every committee member will look at every application in detail (we usually have 100+ apps for any position), and in a lot of cases, it's actually already clear from the key facts that this person is not suitable for the position.

(3) In a first meeting of the committe, every application is discussed individually. As I said, this doesn't take that much time since most cases are obviously a no (mostly because: not the right AOS/AOC - people tend to interpret these requirements very liberally). We then sort them by Yes/Maybe/No. Once we're through the list, we get back to the Maybes and decice whether they're a Yes or a No.

(4) At this stage, we're usually left with 20-30 serious candidates. Depending on the size of the committee, each member will be assigned two or three candidates and do a thorough investigation of their application (each application will be evaluated by two members independently). At this stage, we mostly look at academic writings, especially the PhD thesis and journal papers. Research statements also matter, to see if someone will fit well in the department. Teaching statements, not so much.

(5) In the next meeting, committee members will give a presentation on their candidates and recommend to invite them for an on-camopus interview or not. It usually boils down to 5-10 invites. Sometimes, there are still controversial cases at this stage, and we might discuss your application again in detail, but this is rare.

(6) Final stage: on-campus interview. At this stage, is basically luck and/or chemistry and what you did or did not put in your application doesn't really matter anymore.

[Not sure whether I'm diverse enough]

A question on demography and "not diverse enough." I think I'm a diverse enough person, though I've also been told directly that I'm not the right type of diversity (being Asian male). But here the question is: when someone is invited to interviews as the outcome of the previous list being "not diverse enough," does the diverse person have any chance? Or are they there just to satisfy university admin? Been wondering whether I've been getting interview invitations just because of being diverse.

[associate prof at R1]

@Not sure whether I'm diverse enough: I'm not sure I totally get your question, and I can only speak for my university, but (A) I very much doubt that diversity considerations are playing any role in your interviews; (B) at my university, my admin (for better or worse) does not consider Asian and Asian-American candidates to add anything, diversity-wise, to the pool, so they would not be added in this way (as it would do nothing to assuage the admin--I think this stance is very common at US universities); (C) we (the search committee) do add in underrepresented candidates at the long list stage if they have not gotten in there via the process we've already used (and I think this is pretty common, from talking to other people), but these candidates are not disadvantaged (except by anyone's racism, etc.--which is unfortunately real) going forward in the process--the short list is just the short list (the first round interview list), and all the candidates on it are then treated as viable candidates. For example, we typically (but not always) divide up the candidates so two people read each file initially. But in the short listing process, everyone brings favored candidates to the table and I won't have seen 2/3 of those files yet (in the last search I was in, we had 6 committee members). I wouldn't treat those new files that my colleagues add any differently than I would treat files that were added in any other way; I now just have a new pool (the long list) that we are trying to narrow down to the interview list, and I re-look at each candidate closely in the process. Thus at least for us, if you make it off the long list onto the short (first round interview) list, it's because all of us were extremely impressed with your file, and the only thing that might be different for different candidates is how they got onto the long list, which at that point doesn't matter any more!

[l]

Not sure whether ...
from my experience during a search where the committee was asked to reconsider a file by HR, you really do get a fair shot afterwards. In the case I am thinking of, the person that HR alerted us to was HIRED! I think in our case, the candidate under-sold themselves (I cannot go into details). But we were required by HR to reconsider the file. We did, and the candidate was interviewed. The candidate did very well at that stage. So do not give up, or think that these exercises are predetermined.