What should (not) be in a teaching portfolio?

In the comments section of Trevor's post on teaching portfolios vs. 'evidence of teaching effectiveness', a reader writes:

I'm dramatically downsizing my portfolio, since I've heard from folks now that search committees just don't care about seeing all your syllabi, or sifting through full evaluation sets. For me, this means I'm summarizing all of the data, whether quantitative or written comments, and including just two full syllabi, plus of course a statement. And that is that.

If a Department really wants to see full, unaltered evals, then their language should say exactly that. Or so it seems to me. If they don't ask for that, then they shouldn't expect more; nor should they let biases of cherry picking figure into their assessment if they're not asking for the full set.

But I'd like to hear from Trevor and Marcus on this.

Another commenter, Annie, responded as follows:

I would have thought that 'evidence of teaching excellence' means unaltered evaluations, awards, informal letters of support for students (if you opt to include that sort of thing), reports of teaching observation, etc. To my ear, a teaching statement and syllabi don't fit the bill.

Finally, in response to Annie's comment, another reader wrote in:

I had sort of been wondering about this, and now someone has responded in the post about 'evidence of teaching effectiveness' VS 'teaching portfolio' that the former might involve letters of reference written on your behalf by students. But is this something hiring committees actually like to see? It seems pretty strange to me.

All of these comments raise an interesting question: given that 'teaching portfolio' and 'evidence of teaching effectiveness' seem to be used interchangeably, what should and should not be included in a teaching portfolio?

Here, for what it is worth, is what I included in mine:

1. One-page teaching statement.
2. Multi-year summary overview of student evaluations (including numerical comparisons to college averages).
3. One semester's worth of complete and unedited student evaluations (including all student comments for at least one course).
4. One complete syllabus.
5. Several course outlines (for courses I might teach if hired).

I imagine that including a teaching observation letter or two by a faculty observer might also be fine to include (though I simply had one of my recommendation letters be a 'teaching' letter). However, I have to confess that I'm skeptical that it's a good idea to include student letters/testimonials. Because most of us have at least a few students who seem to love what we do, I have to imagine that members of hiring committees are unlikely to afford much value to a few glowing letters by students. Rather, having talked to search committee members myself, my sense is that they are more interested in the overall tenor of student evaluation comments. Indeed, I know at least one search committee member who thinks that the negative student comments in evaluations are sometimes the most interesting, as far from reflecting negatively on the candidate, they can show that (1) the candidate challenges their students rather than simply handing out high grades, and (2) including unedited comments in the portfolio shows that the candidate is a professional who knows they can't hide from negative feedback.

Beyond the above, I'm not sure what is or is not a good idea to include in a portfolio. I will say that having a super long portfolio (e.g. with PowerPoint lectures, assignments, etc.) seems to me a bad idea. While in principle these kinds of things seem like evidence of teaching effectiveness, I vaguely recall coming across 50-75 page teaching portfolios before and thinking it was just too much. Search committee members are strapped for time, and probably (I think?) won't have the patience to wade through all of that. The teaching portfolio, I see it, should be thought of by candidates as a place to give some idea of what your teaching is like (viz. a good teaching statement and a few other materials). Your task then, if you get a first-round interview and/or on-campus visit, is to show those qualities more directly. 

But these are just my reactions, and as with any single person's reaction, they may be idiosyncratic. What do other readers think, particularly those who have served on search committees? What should and should not be included in a teaching portfolio, and why?

Comments

[teacher]

Marcus
I think teaching portfolios grew too large over the years. Candidates started put everything that might possibly be wanted into them - course syllabi for numerous courses they haven't taught, but would like to, etc. Such large files become useless when a hiring committee has 80 (to 200) applicants.
But I think candidates should think of TWO distinct parts. There is EVIDENCE of effective teaching - this includes, for example, some evaluations, and observations from fellow faculty members. Then there is the various supporting material, like sample syllabi, and a teaching philosophy. I recommend: be selective, and pointed. Personally, I find most statements of teaching philosophy to be pieces of crap. There is little reason to think they really describe what the person does, and when it does do that it usually does not single the candidate out in any way. And there was a point where they became lingo laden with terms like student-centered. Such talk pleases administrators, but they are not reviewing the files, and most of them are far from the classroom.

[Exploiter]

The coup d'incompetence has brought me back to the internet for a few days, and apparently has me feeling like writing a thousand words on teaching portfolios. So here's another response, worth exactly what you paid for it.

First, I think that the terms "teaching portfolio" and "evidence of teaching effectiveness" are interchangeable, and are just two ways of describing the same thing. I sent a similar teaching portfolio to every place I applied whether they asked for a "portfolio," "evidence of teaching effectiveness," or nothing at all.

I also agree with Marcus on several other points as well. I wouldn't include letters by students; I'm not sure why anyone would take those seriously. I would (and did) send completely unedited student evaluations, both quantitative and written. On the written evaluation comments, I think Marcus is right that what people look at is the overall gist of the comments: do you have a lot of "This professor is fantastic! I learned so much! I didn't think I would like philosophy but this course turned out to be my favorite of the semester!"? If you do, then a few random negative ones aren't going to hurt you at all. I had one in mine that derided the white privilege of my early modern class because I hadn't included any African women on the syllabus. I left it in there with all the others. I had at least five or six full pages of written comments from various courses.

In general I think it is a mistake to downsize your portfolio if in doing so it shrinks below 20 pages or so. The reason is the same one for why non-elite grad students need to publish in graduate school: everyone else is doing it. Most other candidates will have long teaching portfolios, and so you need one too. More on this below.

Creating job documents is a difficult thing to do for many reasons, and one of these is that in many cases you cannot go by what people SAY they want to receive. For example, one of the comments in the earlier post, quoted by Marcus, says that they've heard from search committee members that "search committees just don't care about seeing all your syllabi, or sifting through full evaluation sets." This is what people on the committees say, but the key is ignoring what they say and looking at what they DO.

When creating job documents, the question to ask is not, what do search committees say they want? The right question is, how will the search committee actually USE my documents in evaluating my candidacy?

The problem with listening to what search committee say they want is that it gives you an incorrect idea of how they actually judge you. So suppose you've heard from various committee members that they don't want to "sift through full evaluation sets." Fine. But virtually no committee member ever actually does that. If they receive a 70-page portfolio with 30 pages of written comments, they don't say, "No, not another long one!" and then read the whole thing. They just flip through it and look at a few comments here and there, checking to see whether they are mostly positive. And, and this is really important, they do the very same thing with a 50-page portfolio, and one of 40 pages, and one of 30 pages. Their process does not change just because you sent in something longer. They do not and will not read the entire thing, no matter the length.

This is why it's important to focus on their actions and thought processes in creating the documents. To answer the right question about how they use the documents, we must look beyond a job ad to the process a committee member actually follows in evaluating applications. Suppose the committee has 4 people on it and has 300 apps. That's 75 apps per person to evaluate. Each person needs to cut about 65 out to leave the long short list of 40.

How do the committee members make those cuts? By carefully sifting through each aspect of each application? Definitely not. They will use heuristics to make it easier on them. Here are a few they might use:

-Keep everyone from an elite PhD
-Keep everyone with 3+ publications
-Keep everyone who has taught more than 6 solo courses
-Keep everyone who has a letter of rec written by someone I know
-etc.

Obviously there will be others, I'm just trying to give an idea of the process they are using.

Now, finally back to the teaching portfolio. Because they are using these kinds of heuristics to make many judgments in a short period of time, you cannot count on them to look at your materials very carefully. So when they receive a downsized portfolio--I mean one that is substantially shorter than most of the others they have seen--what are they going to think? I suggest that most of them will immediately conclude that you aren't an experienced teacher. They will make this judgment purely on the basis of the page count of your portfolio--they will conclude that you can't be that experienced because if you were, you'd have more material to put in the portfolio.

Those sorts of snap, first-impression judgments can be significant barriers to overcome, depending on the person. You want to avoid creating them in the minds of committee members as much as possible.

On the more positive/cynical side, given that they are making many judgments quickly, they will be prone to cognitive biases that you can exploit. Here's an example of one. If your quantitative evals are a little bit higher than the department average (or whatever baseline you are comparing to), exaggerate the scale of your graph so that it looks like they are MUCH higher. People will pay little attention to the scale of the graph and will make their judgment based on how far one bar surpasses another on the page. If you're a 4.7 and the department average is 4.3, then start the scale at 4.0 or 4.1 so that your performance appears to tower over the average.

Yes, anyone who thinks carefully about it would know that, in the world of student evals, there is no real difference between a 4.7 and a 4.3. But thinking carefully is not what committees do when they make their initial cuts to the application pile. If you make it LOOK like you are a lot better, they will form a judgment that you are an exceptional teacher.

Find the biases, and exploit! Exploit!! Exploit!!!

Relatedly, teaching statements do matter a lot, but not because they convey anything substantial about your teaching (they don't). They matter only as a way of making you stand out. Again, think about how the committee members experience that document. They will read dozens and dozens of statements that are all some form of "blah blah blah student-centered, blah blah blah diversity and inclusion, blah blah blah flipped classroom." But if you are able to come up with an exciting and interesting way of describing your teaching, then you will stand out! They'll know nothing substantive about your teaching but the judgment they'll form is, "This person sounds like a fun and exciting teacher!" And it is that judgment which puts you in contention for one of the 15 first-round interviews of 300 applicants.

[Trystan Goetze]

The simple answer to this problem is for search committees and writers of the ad copy to be more specific about what they actually want in a teaching dossier. Do you want syllabi? Ask for them. Do you want unaltered or summarized evaluations? Be specific. If you don't, you'll get a confused mix of probably too long materials as we all try desperately to distinguish ourselves against unclear criteria. Don't make people play a guessing game and then penalize them for guessing wrong.

It might be helpful if the APA (or a similar organization) could publish a standard format for the teaching dossier (or several standards, to acknowledge the differences between institution types, e.g. research-focused vs. teaching-focused). Individual departments could then pick a standard and add to or delete from it as needed for their purposes and commitments. The standards would have to be flexible enough to accommodate various teaching styles, but there are enough common elements (teaching statements, diversity statements, syllabi, lesson plans, observation letters, etc.) that it should be doable.

A potential side effect might be that candidates missing certain common elements of a standard dossier might be disadvantaged, such as British PhD students who might not have any syllabi for previously taught classes because they often never have a chance to teach as lead instructor. But they would be (and are!) disadvantaged anyway. At least with clear standards, they would know in what way they are disadvantaged and what they might do to fill in their weak points.

[Trevor Hedberg]

It sounds to me like Marcus included roughly the same materials in his teaching portfolio that I did -- that is, on the occasions a full portfolio was requested. I can't imagine compiling a 50+ page portfolio: during the last cycle, mine was only 29 pages, and the first 4 were the most important. These contained my teaching statement and a 5-year numerical summary of my student evaluations. Echoing the comments of a few other people, I've not received any advice to include letters of recommendation from former undergrad students -- better to have one of you faculty letter-writers fill that role.

I agree with Trystan that the whole process is much easier when search committees tell you straightforwardly what they want, but that doesn't always happen. In those cases, I think you have to submit the essentials (i.e., a teaching statement, a summary of teaching evaluations, and at least one sample syllabus) plus anything that's particularly relevant for the advertisement posted.

[Nomad]

I randomly came across this question. I would like to add my experience to this because I am definitely on the minority side of this and may provide a more balanced perspective. In short, my teaching portfolio includes only my teaching statement, a sample syllabus and a letter from a faculty member. Sometimes I didn't even submit teaching portfolio at all (e.g. when applying to Syracuse). But I still got half of the interviews from TT positions, including the places I didn't send teaching portfolio to at all. I honestly think that if I were to include more teaching documents that are excellent, I wouldn't do much better. (In the end, I was also lucky enough to get a position I was happy with.) Of course, this is not to suggest that your better prepared portfolio won't pay off, but perhaps to take off a bit stress over this.

But I do think you shouldn't include anything bad. In one application, I included the most recent evaluation qua requirement, which was not good. And I got rejected fast.