The Secret Lives of Search Committees - Part 13: teaching dossiers

I have talked about teaching statements and teaching portfolios several times in two previous series. I still stand by the things I wrote there, particularly about how important it is for teaching statements to use concrete examples of teaching practices rather than broad generalizations and emotive language. However, my recent experiences on search committees and discussion with other people who have served on search committees elsewhere have driven home a few other important things I think candidates should know:

Too many teaching portfolios are indistinguishable: As I mentioned here and here, one of the most important things candidates need to do is "stand out" from the crowd. Your dossier is one of dozens or even hundreds of others the search committee is looking at. Particularly if you are applying for a job at a teaching school, the committee is looking for something that distinguishes you as a teacher - something that makes you "jump off the page" as someone particularly worth interviewing. Alas, in my experience, roughly 8 in 10 teaching portfolios are more or less indistinguishable from one another, saying similar things (such as stating that they diagram arguments, have students do group work, engage in Socratic dialogue, etc.), having similar teaching evaluations (basically, fairly happy students), similar looking syllabi, and so on. Unfortunately, it's a simple fact: if you're doing what everyone else is doing, you're going to have a hard time standing out. So, either you need to develop a more original pedagogy, or you have to show in some clear way how you are particularly good at what you do. In the teaching statement, concrete examples of in-class discussions, assignments, etc. are helpful, as in my experience hardly anyone actually does things the same way in the classroom (even those who do Socratic dialogue typically do so in distinct ways). On a similar note, I think it's important to not only include syllabi in the teaching portfolio, but also examples of term-paper assignments, rubrics, etc.

Programs are under-training teachers: I've said this before recently, but it bears saying again. My sense is that many programs are seriously under-training candidates as teachers for jobs at institutions like mine. It's not just about the amount of teaching experience that's an issue (though it is). Another problem is that too many teaching statements boil down to something like this: "I stand in front of the classroom and talk about philosophy with students." I suspect there's a reason for this. In all of my years in grad school (in two PhD programs), we had proseminars and even a "professionalization" seminar (where we practiced presenting our research, crafting dossier materials, etc.). However, to the best of my recollection I never once had a teaching seminar - i.e. any kind of formal or informal setting to discuss or experiment with pedagogy. From what I can tell, many programs "train" their candidates to teach by sticking them in front of the classroom and maybe having a faculty member sit in and observe them on occasion. The end result is candidates on the market who are not only under-experienced but also under-trained with the experience they do have. People at institutions like many take pedagogy very seriously. Grad programs and candidates need to do so too if they want to be competitive for jobs at teaching schools.

Teaching statements should be one page: I was first told this by the job-market consultant I used, have heard more than a few other people who have served on search committees say the same thing...and having served on three search committees, I agree. Search committee members have hundreds of applications to read, constituting literally thousands of pages. It is exhausting work, and there are few things worse than reading a teaching statement that takes forever to get to the point. If your teaching statement is two or three dense pages long, but some other candidate has a pristine one page succinctly demonstrating what makes them unique and interesting as a teacher, they will almost certainly have an advantage. A beautiful one-page teaching statement in a pile of verbose 2-3 page ones is a real breath of fresh air. It's hard work condensing a teaching statement into one page, but it can be done.

Too many teaching portfolios appear thrown together: I'll be honest. I've looked at a lot of teaching portfolios, both as a search committee member and as a mentor. Far too many of them look thrown together haphazardly, like a teaching statement followed by photocopies of a set of course evaluations, followed by maybe a syllabus. This looks bad for a number of reasons. First, it can just come off lazy. As I've mentioned before, people at teaching schools are looking to hire "professionals" - people who really have their act together, showing polish and conscientiousness. There are many reasons why this is important, not the least of which is that people eventually have to put tenure files together, reports for deans and committees, and so on. A candidate who has put together a well-0rganized and presented portfolio (with an actual  table of contents, concise summary of quantitative data, etc.) is going to look better, all things being equal, than one who doesn't. Second, haphazard teaching portfolios obscure more than they illuminate, and can irritate a reader by making their job difficult. Take for example raw student evaluations: I assume we all know how cluttered those things can be, with tons of questions, department and college means, and so on. Having to read through raw photocopies of student evaluations (including illegible student handwriting) is not only irritating but confusing. Wise candidates should instead provide summaries of student evaluations and typed student comments.

Don't provide "select" student comments or course evaluations: I've had a number of friends who have served on search committees complain about this. As one of my acquaintances said (I paraphrase), "If I come across a teaching portfolio with 'select' student comments, I immediately wonder what the candidate is trying to hide. So I may look up their ratemyprofessors page to find out their students think they are the Worst Teacher Ever. And, all too often, that's exactly what I find: a litany of wretched RMP ratings that make the candidate look really bad. The same is true if the person only includes course evaluations from years ago, like 2012 or 2014. Once again, I think to myself, why didn't they provide more recent evaluations..."

Look, regardless of whatever qualms one may have with search committees looking up ratemyprofessors pages, my sense is that it happens, and is far more likely to happen if you try to hide negative student comments or courses with poor quantitative data. Further, I've heard a few search committee members say that just trying to hide these things reflects poorly. Given that one cannot hide from negative student reviews once one is hired, trying to hide them in the application process can look deceptive or at least out of touch with the fact that one cannot hide from inconvenient things as a professional.

So, then, what's a candidate to do? Fortunately, here's my other sense: there's an increasing number of people who doubt the value of student evaluations, and for good reason. My advice, then, is: don't try to hide from negative comments or evals. You're better off including stuff in your portfolio so the person reading it at least gets a balanced view of what your students think. I think this is especially true if you're a challenging instructor. The person reading your application may know that negative student evaluations are strongly predicted by difficulty, not anything wrong with one as a teacher. For what it is worth, this may be my favorite scatterplot ever:

Difficulty matters: Following up on my point above about student evaluations, my sense is that people on the hiring side of things at teaching schools are not just looking for teachers who are well-liked. Although a high level of student enmity may be off-putting to search committees (who at teaching schools often depend on majors numbers for their department to remain viable), they can also be put off by the "easy teacher." I don't know anyone who has served on a search committee at a teaching school who would say they want to hire someone whose students like them because they are easy. No, my clear sense is that people at teaching schools tend to be looking for people who have the integrity and courage to be challenging instructors who nevertheless know how to engage students. Consequently, the questions in your student evals that might actually mean the most are the questions on course difficulty and assignment difficulty.

Anyway, these are just my thoughts. Perhaps my experience, and the people I've spoken to whose thoughts informed these comments, are idiosyncratic. I'd be curious to hear what others who have served on hiring committees think, but hope you all find these thoughts helpful!

Comments

[Daniel Brunson]

For those with years and years of teaching evaluations (dozens of courses), would you recommend a quantitative summary of all evaluations, and then a full quantitative and qualitative evaluation for a (most) recent course or semester?

[SLAC Associate]

I entirely agree on the point about not including selected student comments; selected comments are practically worse information than no comments at all. (Nearly) every applicant has a few students who majored in philosophy because of their class and that think they're the most supportive professor ever, etc., etc. I want to know what the other 90% of the students in your class say; the content of the middle-of-the-road comments often tells me more about an applicant's classroom than anything else in their file.

Also, including a sample assignment or two in the teaching portfolio can also be a way of distinguishing oneself, especially if the assignment is relatively unique and interesting and not just another essay comparing the Apology and the Crito or whatever.

[Michel]

I'd be interested to know what people consider a good length to be for the portfolio as a whole. 40-50 pages? More? Less?

[Amanda]

I picked 4 courses, representative of what I taught, and included all student comments, typed, and an easy to read summary of the numberrs. It didn't make sense to me to include every class I taught, as no one has time to read it. I think my portfolio was 10 pages total. Worked for me. Anway, I agree with this advice.

[Marcus Arvan]

Hi Michel: My teaching portfolio was about 30 pages. Here's a link to the one I used: https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsdGRvbWFpbnxwaGlsb3NvcGhlcnNjb2Nvb24yMDE1fGd4OjVmNTBmNDJjNTQ4OGQ4OTg

I was told it was good to include (i) a teaching statement, (ii) quantitative summary of recent evaluations, (iii) a few full sets of evaluations including complete and unedited student comments, (iv) several full course outlines, and (v) one full syllabus. I think this is about right for a teaching job. In retrospect, I think I could have also added some assignments - which could have brought my portfolio to 35 pages easily.

I don't think this is too much if the portfolio is well-organized - though, as Amanda said, her 10 pages seemed to work fine for her.

[Marcus Arvan]

Daniel: good question. I would advise something like what I did in my portfolio - a quantitative summary spanning several years, followed by full course evaluations for a semester or two (including complete student comments for at least one course).

[Amanda]

To clarify what was in my portfolio: one page teaching statement, summary of qualitative evaluations for four classes, complete set of unedited comments for four classes, 3 course outlines and one full syllabus. The assignments were on my syllabus, and I gave examples of class activities in my teaching statements. I keep my syllabi short, for a number of what I like to think are justified reasons.

[still on the market though :(]

Regarding lack of teaching prep coming out of grad school, I just wanted to recommend/give a shoutout to the American Association of Philosophy Teachers (AAPT). I attended their seminar for grad students and some of their workshops, and they were so helpful not just for shaping and shifting my pedagogy but also helping me to more clearly articulate my pedagogy. (And post-grad school I still find their workshop-conference and the lasting friendships I’ve formed there helpful as my pedagogy evolves, as I run into new challenges, as I start feeling stagnant, etc.) I know there is some privilege involved in being able to travel and attend, but there has sometimes been some funding available to help grad students get there. And they sometimes put on workshops in different areas.

[Lauren]

I was lucky to have a department with a pedagogy seminar as well as an active teaching center with pedagogy courses for grad students, and I also took advantage of an AAPT seminar. These were all so helpful in helping me to think carefully and critically about my pedagogy and to be able to talk about it (both in writing and in interviews). I might push back on Marcus' suggestion of sticking to one page only for the teaching philosophy, though--success is anecdotal, of course, but I was pretty successful with a 2 page teaching philosophy, and I got a lot of compliments on it from places I interviewed. I think if I'd had less substantive and interesting things to say, it might have been too much, but mine really was organized around a particular approach to teaching, with lots of concrete examples. But I will say that I when I hear grad students say they think the teaching philosophy is the easiest document to write, I shake my head--I found it really difficult to write a good one and I revised mine countless times.

[elisa freschi]

1. How can you avoid selecting *some* students' evaluations if you have been teaching for, say, 10 years?
2. If you type in the students' comments, could not you also "edit" (i.e. falsify) them?

I apologise for my naïve questions (as mentioned several times, I come from a world where student evaluations are never part of one's dossier).

[Amanda]

I think the idea, Elisa, is to have all student comments from a single class. Like I only had four classes but I included all comments (not just select ones) from those classes. And yeah, I do think it would be really easy to falsify them. I think we are working on the honor system here. Some places ask to see raw evals - but usually not. And when they do I honestly doubt they check.

[SLAC Associate]

To elisa's questions:
(1) My preference is to see all student comments from one semester or one year. Preferably they would be from the most recent year of teaching, though there could be reasons to select a slightly older set of comments with sufficient explanation: "I've included the most recent set of data with response rates over 80%"; "I've included the most recent set of data from course XYZ that was specified in the job posting," etc. What I think ought to be avoided is presenting 5-10 of the most positive comments from across one's entire teaching career while leaving out any sense of how representative those comments are, or what the remainder of one's students seem to have thought.

(2) One could falsify the data in their application, but that could be grounds for voiding their contract after being hired if the search committee ever asked for the originals. (Compare: an applicant could also lie about already having the PhD in hand, or lie about forthcoming publications, etc. But I think it's obvious that doing so could seriously endanger one's future career prospects.)

[Marcus Arvan]

Hi Elisa: Thanks for your queries.

The point I was trying to make in (1) is to not choose courses selectively to make yourself look good (i.e. choosing a course from 2013 and another from 2016 because they were both good sets of evaluations). Rather, one should provide one set of evaluations for a single semester or year, so it doesn't look like you're just cherry-picking evals that look good. For instance, if you look at my portfolio I linked to above, I included actual evaluations for a full academic year. Some of my evals were far higher than my college averages while some hovered around those averages. That gives the reader a balanced picture of what my students thought of my, not just a cherry-picked presentation of what my "best" evaluations were.

(2) In terms of lying, I agree with SLAC - it's possible, but the risks are so enormous that I would hope few (if any) candidates would do it. If someone submitted "unedited" student comments but they appeared suspicious (viz. lacking any negative feedback), a committee might request the originals for verification. If they were found to have falsified comments, a voided contract might be the least of their concerns: they could be brought up on charges of academic dishonesty at their institution, which might ruin their chances for a job anywhere.

[still on the market]

In my portfolio I include one page of selected comments to give a sense of what my particular strengths are in the classroom, and then I include complete data (quantitative and qualitative) for a single course, or in the past I have included all comments for a single semester.

In this case, will including a page of selected comments in addition to complete comments for a course or semester still be viewed negatively by many committee members? Should I omit the page of selected comments? The other part of my reasoning for including the page of selected comments is that if other candidates are doing it, even if some committee members view these as useless, it's hard to believe that these positive comments won't have at least some unconscious effect on the readers, and I am competing against people who are including these.

[Marcus Arvan]

still on the market: I would just include complete and unedited comments, not the selected ones. I don't think the positive comments alone will have the effect you desire. As SLAC Associate notes above, people on the hiring side are *not* the most interested in what your happiest students think of you. They are interested primarily in what the full-range of students think: primarily the people in the middle who neither love nor despise you as a teacher.

My sense is that any attempt to put a "positive spin" on evals by including selected comments seems to me liable to backfire for this reason alone - it just has the wrong *feel* to it. People on the hiring side of things are looking for candidates who don't try to hide or downplay bad or mediocre stuff. You certainly don't get to downplay it when you come up for mid-tenure or tenure review: you have to face up to your complete record. Consequently, my sense is that it looks good when candidates don't look like they're trying to suppress negative information. It indicates they are a real professional who knows they can't hide from things, and a person who has the integrity to present themselves as they are.

[SLAC Associate]

One thing I've seen several candidates do is provide complete and unedited comments, but bold those to which they wish to draw the committee's attention. That strikes me as a better way of doing what "still on the market" was suggesting, especially if the bolded comments are not just all the most positive ones but the ones that you think are most representative of your teaching. (Doing this probably requires typing up the comments yourself, but depending on how legible your official student evaluations are, you may already have reason to do that.)

[still on the market]

Thanks Marcus and SLAC, this is really helpful. My other reasoning for including the selected comments page in addition is that I teach around 120+ students per semester and have a fairly high response rate so it ends up being just a ton of comments for search committee members to wade through. I see how bolding some comments could help, but I wonder if this will still rub some committee members the wrong way, even if I bold some of the critical comments also.

Anyway, my evaluations tend to be quite good, so this wasn't the reason in my case for adding the selected comments page--rather it's just something I thought would be helpful. At any rate, it is good to know that taking this page out won't count against me.

[still on the market]

OK, one other CV question. I am three years out from my phd and have only one publication, which is several years old. I have two articles almost finished that I am trying to get under review. I know there is no good excuse for this.

Multiple senior philosophers suggested that I include a Works in Preparation on my cv to help show that I really do have an active research agenda. (As a side note, I am extremely teaching focused but happy to do whatever kind or amount of research will help me get into basically any teaching position.)

I understand that it is frowned upon to put works in progress under one's publication heading on one's cv. But is it really so frowned upon to put these under a separate sub-heading, which is clearly marked? I am clearly not trying to be deceptive and clearly understand the difference between published papers and unpublished drafts--I just want my cv to be neat, concise, and readable.

[still on the market]

sorry, that last question should have gone on the cover letter post, which mentions this issue in cvs

[Marcus Arvan]

Okay, I’ll repost it over there on your behalf and respond over there.

[Marcus Arvan]

"I have two articles almost finished that I am trying to get under review. I know there is no good excuse for this.

Multiple senior philosophers suggested that I include a Works in Preparation on my cv to help show that I really do have an active research agenda."

still on the market: I'm going to reply to this both over here and over there because I think it is a really important issue (I'm also going to address it in an upcoming post on CVs).

I have bad news. I don't think a Works in Preparation list is going to help. People on the hiring side of things who care about research (including people at schools like mine) want to know that you can publish more than once - specifically, that you know how to publish well-enough to get tenure. A Works in Preparation page simply does not demonstrate that, and so I do not think it will confer any advantage.

The only thing that will confer an advantage on you research-wise is to publish. Fortunately, I think I may be able to help with that. Why haven't you been able to publish? If it's because you've primarily been sending stuff to highly-ranked journals (with high rejection-rates)--as many people are counseled to do--my advice is to *change* your approach.

Like many people, I was adamantly counseled by people who thought they knew what they were talking about that "you shouldn't send stuff to bad journals." Fortunately, I'm not much for simply accepting what people tell me. I started sending stuff to lower-ranked journals and published a ton of stuff. Then my number of interviews dramatically increased, and I got job offers and a job.

Let me be plain: there are places that will count publications in "bad journals" against you - R1's, R2's, and elite SLACs. Unfortunately, if you haven't published more than one thing in a highly-ranked journal after several years, you're probably not in the running for those jobs (unless multiple acceptances from top-journals suddenly come your way). That leaves non-elite SLACs and CCs for you. Do they care about journal rankings. The answer is NO. So start publishing, even in journals the hoity-toity people in our profession look down on. People hiring at schools like mine don't look down on it. It's roughly 1000x better than publishing nothing.

[Recent grad]

What if the response rates for your student evaluations are low? I traditionally haven't pestered students to fill them out--I generally think they're bunk as a measure of teaching quality and I use other methods to gather student feedback. As a result, only 20-30 % respond. I know I can pester them in the future, but if I decide to apply out next year, what should I do about the evaluations part of my dossier?

[Joe]

"As I mentioned here and here, one of the most important things candidates need to do is "stand out" from the crowd. Your dossier is one of dozens or even hundreds of others the search committee is looking at. Particularly if you are applying for a job at a teaching school, the committee is looking for something that distinguishes you as a teacher - something that makes you "jump off the page" as someone particularly worth interviewing."

I'm going to swim against the tide here: I think that this is a really, really bad requirement that search committees should actively seek to avoid. The issue is that good teaching is not necessarily pedagogically innovative: far from it. If someone writes" I use the Socratic method," you may have read this so many times that it makes your head explode. But look at their evaluations: suppose they're brilliant, full of praise and stories of personal growth and all those good things. Maybe this person is using... oh, I don't know... the Socratic Method? Which is a really, really awesome teaching method to which we philosophers have special access? A method which is so important, influential and famous that it has maintained its name for 2500 years?

This is one of the truly bad things about the market: the number of applications makes us look for the unusual and sexy, which is absolutely not well-correlated with the responsible and effective. If this is how you use teaching statements, then stop requiring teaching statements and save yourselves and everyone else a whole bunch of time.

[Marcus Arvan]

Hi Joe: my department hired someone this year who uses the Socratic method - so I think you’re right, it can be done incredibly well and search committees should not favor “different for the sake of being different.” We hired the person we did this year for the reasons you give: they are clearly an outstanding teacher! But let me tell you one thing, in full honesty. The problem isn’t that 9 out of 10 candidates use the Socratic method. The problem is all too many candidates some across as though they have never given teaching much thought. There’s a big difference between being able to demonstrate that you are *fantastic* at traditional teaching method—which does make one stand out—versus coming across as someone for whom teaching does not look like a priority. When I say it’s important to stand out, that’s what I mean. Being a creative teacher is one way to do it, being an excellent Socratic teacher another - but you have to show that there’s something about you that distinguishes you from the 150 other candidates whose teaching dossiers blend into each other. My message is: this is very hard to do, and people who want to be successful on the market for teaching jobs need to find a way to make sure they don’t come across as just another average teacher.

[Amanda]

Recent grad - yes, pester students - or find some way to bring up your response rate. I, in fact, do achieve this by polite pestering. And for now - you can only do what you can do. Simply state these are your complete evaluations, and note the response rate. I would not bother going into any justifications for why the response rate was low, as those almost always make you look bad. Instead just state it and let the cards fall where they may, as you don't have other options.

[A]

What do people think about including thank you emails from students in the teaching dossier? Some of those emails include a detailed (positive) evaluation.

[elisa freschi]

Many thanks to Marcus, Amanda and "SLAC associate" for their clarifications.