Job-Market Boot Camp, part 28: common mistakes with teaching portfolios

In my previous two posts in this series, I presented what I take to be some common mistakes in teaching and research statements. I hope some more search-committee members weigh in, so that job-marketeers can hear other perspectives. But in any case, in today's post I'd like to discuss some common mistakes in teaching portfolios. For those less well-acquainted with teaching portfolios (e.g. new or future job-candidates), teaching portfolios are perhaps the most open of all job-market materials. Although they standardly include things like student evaluations and sample syllabi, some portfolios including much more, ranging from examples of assignments to sample lectures.

I myself don't have any settled views about exactly what teaching portolios should include--so I'm very curious to hear in the comment section below from other search committee members about what they think should or shouldn't be included, or how long teaching portfolios should be (particularly given that I've heard some people have teaching portfolios upwards of 50 pages, which seems to me too long). But, for now, let me offer a short list of what I take some 'common mistakes' to be:

Portfolios that look hastily thrown together: which can result from several converging sources of evidence...

1. 1. No table of contents.
   2. Inconsistent formatting.
   3. Overly-short, uninformative syllabi: although I've heard it can be a good idea to provide several course outlines (consisting merely of a course title, course description, and reading schedule), I think it is probably very important to include one full syllabus in a teaching portfolio, and for the syllabus to be detailed regarding course policies, assignments, and so on.
   4. Merely including raw student-evaluation reports: As people have noted on this blog before, search-committee members can be really strapped for time. They typically have hundreds of applications to wade through--so anything that makes their job easier is, prima facie, good to do. Further, as I am sure many people who have served on search-committees can attest, raw student-evaluation reports can be confusing to wade through. They can be full of all kinds of numbers and unfamiliar questions (my own university's student evaluations have 36 separate items!). Consequently, although I think it is good to provide some raw reports (so as to, again, not look like you are 'hiding anything'), I think it is again a really good idea to include some kind of overview summary to make things easier on your reader.

Only providing 'selected' student comments: By my estimation, this may be the single biggest common mistake candidates make with teaching portfolios. My sense is that it may be a serious mistake in a number of different respects. First, it may be a mistake in terms of what it projects: namely, that the candidate may be 'trying to hide something'. Even if you're not trying to hide anything, looking like you may be can be the wrong thing to project--because here's the simple fact: if and when you are hired, you will not be able to hide anything. If you get some poor student reviews or vicious student comments, you're not going to be able to hide from them in annual evaluations, let alone tenure and promotion. Consequently, I think it is important to project that you know this: that you recognize that you can't hide from things as a professional. Second, my sense is that providing 'selected' student comments is likely to be practically counterproductive. For, although I can only speculate, I suspect that what may do is lead search committee members to try to figure out what you are leaving out--by, for example, leading them to look at your Ratemyprofessors page (which, as I assume we all know, can be a place where particularly disgruntled students take out their frustrations). For these reasons, I strongly suggest that if you include student comments at all, include complete and unedited ones, for at least one class. If you are not comfortable with doing that, I'd suggest not including student comments at all.

No longitudinal summary of student evaluations: Everyone can have a 'good semester' of student evaluations, and everyone can have a bad one. For my part, I know more than a few people who don't think student evaluations have much value at all, given the empirical research. Still, of those I know who do care about student evaluations, the thing they tend to care about is consistency. Does a person consistently receive poor reviews or good ones? If a search committee thinks student evaluations measure anything of importance at all, this sort of longitudinal data seems to me important to include. I think it may also make a candidate come across as conscientious--showing that they took the time to collate and present their evaluations in an easily digestible way. 

Absence of syllabi for job ad AOC's: Having served on several search-committees, I know how important the Areas of Concentration a job-ad lists can be. At teaching-focused institutions in particular, hiring committees may care a great deal about hiring someone who has solo-teaching experience in the AOC listed in the ad. While it may not be a 'kiss of death' not to include a syllabus related to the ad's AOCs, my sense is that, all things being equal, it is better to include them--as search committee members can care not only about experience teaching experience, but also how you teach it.

Controversial course policies (particularly) without explanation: This one hadn't occurred to me until someone (Amanda, I think) recently mentioned it, but I think controversial course policies (such as forbidding computers in the classroom) could potentially work against candidates. Although I think it is important to be honest in one's teaching portfolio, it is probably also important to recognize that some search-committee members may have strong objections to certain policies. So, if you do include controversial policies in your syllabi, it may make sense to explain them and/or note exceptions.

Insufficient evidence of assignments and assessments: I didn't include assignments in my teaching portfolio. However, my syllabi contain fairly explicit descriptions of assignments--their structure, pedagogical aims, and how they are assessed. As I mentioned in my previous post on teaching-statements, my experience (particularly recently) is that people can care a great deal about assessment--so I think it is probably important to include something to this effect in a good portfolio.

To be clear, these are just my own reflections on what some common teaching-portfolio mistakes may be. I could be wrong, and if so I'd love to hear search-committee members weigh in with their own thoughts. I am also very curious to hear whether there are other 'common mistakes' I have neglected. Anyone have any insight they are willing to share?

Comments

[Daniel]

It has been a little while since I was on the market, but if I recall correctly, I provided selected student comments but (right there with the selected comments) noted that full comments are available on my website for download and also via request. I assumed that this would contravene the impression that I'm trying to hide something, because if I were trying to hide something I wouldn't put them all available online for anyone to download. Moreover, it made the portfolio a much more manageable length and more readable, since complete comments from even one class would take up a lot of space. Thoughts on this sort of approach?

[postdoc]

Echoing Daniel's comment, since I was confused by the same part of the post. I've seen plenty of dossiers with both a selection of top comments and raw evaluations complete with comments for several courses. Surely an approach like this doesn't come off as though the candidate is trying to hide something? I take it that providing selected student comments would only be problematic if that where all the applicant included in their dossier.

[Marcus Arvan]

Daniel: I think that is a fine idea! As long as you draw attention to the fact that complete and unedited comments are available on your website or by request, it shouldn’t look like you are trying to hide something.

Postdoc: thanks I’ll clarify the post. I should have said that *only* including raw numerical evals and select comments is the issue (and it does seem to me to be a pretty common mistake). As long as complete and unedited comments are somewhere in the portfolio or otherwise available (as Daniel’s approach does), I think candidates are fine.

[Amanda]

I strongly agree that including only the raw, unedited evals is a big mistake. It is often impossible to read, and meaningless without really understanding the context. Canidadtes should (1) summarize their numbers for each class in a way that is easy to read. For example,

*Below is a list of my average score for each of the mentioned questions. My university scores on a system of 1 to 7, with 7 being excellent. (if your university includes too many questions, only include the most important 6 or 8, and always include the "overall evaluation of the course/instructor questions."

While it is fine to include the raw data at the back of the portfolio, I also think it is just as good to include a link to this data, since it takes up so much space and few search committees will look at it initially.

I have mixed feelings about what Marcus says about select comments. I personally would not want to see select comments, but I know many people who have been successful on the market with this method, and many search committee members who don't care. So if you have some very negative comments, it might be best (strategically) to do select comments. This seems especially so if all it takes to satisfy the "all comments included" search committee member is a link to a complete set of comments. Most search committee members won't bother looking, so what will be in their head is whatever comments you included.

A table of contents that makes everything easy to read is very important.

[anon]

Question then:

If I decide to include both positive and negative student comments, this would seem to ward off the impression that I am hiding something, right?

But at the same time, including negative comments might also hurt me? But presumably everyone has negative comments, so maybe not?

So I wonder if the best approach is just to not include any comments at all and just go with quantitative data but make sure you indicate that full data is available upon request?

[Marcus Arvan]

anon: I think the answer probably depends on what your student eval comments are like.

I included all of my student comments because the vast majority of them were positive, and I reasoned (correctly, I think) that search-committee members would look at the big picture. My sense is that it is usually possible to find at least *one* recent class one has taught where the student comments are positive on balance--so all things being equal that is what I would suggest: including all student comments for your best recent class.

On the other hand, if you don't have a single recent class with comments that you are comfortable sharing complete and unedited, then I think the best thing to do is to probably only include quantitative information.

[guy who was looking for a job]

Kind of a n00b observation here, but one thing I noticed last year was that some teaching portfolios I saw were fairly short, as in, only around 8-10 pages. I did not submit one that was shorter than 24 pages, and some of mine were as long as 30 (and I've seen ones that are longer than that from other people). Part of the issue here is that having a thicker teaching portfolio is going to make you seem more experienced, as long as the stuff in there isn't filler. And student evals, either written or quantitative, are not filler.

For what it's worth, I had 4 pages of written evals, and 4 pages of quantitative evals. For the written part, I included every single comment I had received for the courses I included, and noted that the comments were "complete and unedited." This meant that I included a few comments which blasted me for various things, but I don't think that matters. I doubt the committee members who looked at my portfolio read many of the comments. It's much more likely that they are going to skim through a few, see that they are positive (since most of them are), and then just move on. (You can be strategic about the order you put them in, even when including all of them--the very first written comment listed on mine was, "Absolutely take the class with [my name].")

For quantitative evals, again, a n00b observation here, but I have been surprised at how many people just include a list of numbers. This is *not* a good way to present data! You must control the story that you want your numbers to tell. Quit being lazy and design some attractive-looking bar charts or other graphs that will represent your numbers in a format that is much easier to understand.

And again, making the charts allows you to be strategic about the way you present it. As any good scientist knows, you can play with the axes on the charts to exaggerate differences between you and a comparison group (assuming that you are better). It is well understood by now that people are very prone to cognitive biases in situations like this, since the way most people read the chart is by looking at the distance between the bars, without paying much attention to what the scale is, or where the axis begins. This way, you can make a two-tenths difference in mean values seem very large indeed. Of course we can all agree that means separated by two-tenths of a point in teaching eval scores are basically the same; the error bars would be huge, were you to include them. But again, this is not how people read the charts.

I guess I would echo something that Amanda said in an earlier thread: you must imagine that you are the person reading your portfolio, and then you must present the material in the way that will make YOU look the best. And this includes exploiting every cognitive bias of theirs that you can along the way.

[Amanda]

I am not sure a "thicker" portfolio will make someone look more experienced. First, almost everything is online, so the physical thickness will not be apparent in an obvious sort of way. Second, it is just so easy to look at someone's CV and see how many classes they've taught. This is typically noticed before delving into the teaching portfolio, so I suspect search committee members have a picture of how experienced a candidate happens to be before they look at the details of the portfolio.

Exploiting cognitive biases in the way "guy who was looking for a job" suggests might be helpful, all things considered equal. Whether it is worth it, strategically, seems to depend on how long it would take you to put this together, and how much of an increased benefit would result.

FWIW, my got to format was to always include evals from 4 courses. I thought this was more than enough to give a picture of who I was as a teacher, and not too much as to overwhelm search committee members. it is relevant, though, that showing these 4 courses evals conisted in, (1) summarizing the scores from 6-8 numerical ranking questions, and (2) including complete, unedited comments. Also key: I never taught classes with more than 40 students, so the complete list of comments was pretty short. If you teach giant courses, I wouldn't have all evals be from giant courses (hopefully you also taught some smaller ones.) If you have only taught giant courses, than I wouldn't include more than 3 full sets of evals. This is just my best guess on what is likely to show a full picture of your teaching without overwhelming search committee members. Lastly, make sure the evals you choose are recent, i.e. that unless you have been on teaching leave, all should be from the last 4 years, and at least one from the most recent year