Notes from both sides of the market, part 3: teaching portfolio dilemmas

As I mentioned in my post on teaching portfolios in last year's Job Market Boot Camp, I tend to think it is a good idea to include:

1. An extended summary of raw quantitative student evaluation data spanning several semesters.
2. Complete and unedited student comments from several courses and/or semesters.

The general rationale for both of these suggestions was that, especially at teaching schools, search committee members may be interested in a complete picture of a candidate's teaching experience, including whether the candidate has a consistent or spotty record of "student satisfaction." Now, of course, I realize that student evaluations themselves appear to have a spotty track-record. Still, for all that, for better or worse, student evaluations appear to matter in today's higher-education environment. First, at least in part because of recent financial struggles in public higher education and smaller private schools [some of which have closed down or closed "underperforming" departments], departmental funding--and indeed, departmental viability--can depend on student enrollment and majors of numbers, both of which are probably related to "student satisfaction." Second, again for better or worse, student satisfaction plausibly plays a role in tenure decisions at teaching schools. Consequently, or so I'm inclined to think, there are reasons to think it is important to provide search committees with a clear and complete picture of one's student evaluation history: not only one's good evaluations, but also one's bad ones.

At the same time, I recognize there is a certain dilemma here--one that I have encountered in a number of different ways, including in my experience as a job-candidate mentor. Broadly speaking, the dilemma is: what should one do if one has a particularly spotty student evaluation record? On the one hand, one could include all of one's quantitative data and student comments, which runs the risk of showing search committees less-than-flattering information. On the other hand, one could provide a more limited record of student data and comments, such as:

1. A document including only "select student comments", disclosing primarily positive student feedback.
2. Complete quantitative information for only one course, or one semester of courses, in which one received high scores.

I have seen more than a few people adopt this approach. Yet, while it omits negative information and highlights the positive, it also intuitively runs several related risks. First, it could raise a "red flag" to search committees that there is probably negative information the candidate didn't disclose [which could, among other things, leave them wondering just how bad the omitted information might be--viz. does the candidate merely have some less-than-flattering comments, or a litany of student comments alleging incompetence, etc.?]. Second, it could plausibly lead search committee members to have concerns about the candidate's integrity--for, while leaving out negative information is in one sense understandable [no one likes displaying negative information about themselves], it is also deceptive, intentionally withholding information that hiring committees and administrators might consider relevant and important. Third, it could also lead search committee members to wonder about the candidate's understanding of annual evaluations, tenure, and promotion--for, if a candidate is hired, they will ultimately be evaluated not on a "select" presentation of their teaching and research record at the institution, but rather on their full and complete record.

So, then, what should a candidate facing this kind of dilemma do? Truth be told, I am not entirely sure--and so would like to ask you all, especially those of you who have served on search committees: is it better for a candidate with a "spotty" student evaluation record to disclose a complete and unedited record of quantitative data and student comments in their teaching portfolio, or, is it better for the candidate to "impression manage" their record [e.g. providing only a small sample of course data and/or "select" student comments]?

For my part, I guess I am inclined to say that it's better to provide a full record, and for several reasons. First, it just seems more honest--and I wouldn't be surprised if search committee members tend to appreciate honesty and full disclosure [if one is hiring, it can plausibly be frustrating to feel as though one does not have full information, especially if a candidate seems intent on hiding some of it]. Second, it plausibly reflects some amount of courage on a candidate's part--as I suspect anyone can recognize that it is not easy to disclose unflattering information [and, since courage is generally a virtue, this might reflect positively on a candidate]. Third, since [or so I'm inclined to think] most people who work hard at teaching tend to improve over time, providing full information might provide a clearer picture of an overall positive "teaching narrative" [viz. maybe the candidate still receives some negative comments, but there is a clear record of improvement over time]. Still, at the same time, I recognize there are potential risks to full disclosure [what if some student comments are particularly brutal, claiming a person is incompetent, unhelpful, etc., as a teacher?]. However, again, I cannot help but wonder whether it is better to deal with these issues openly--for instance, in one's teaching statement, in which one might address past teaching struggles and frame oneself as a committed teacher working hard to improve student perception.

Finally, I think it is probably worth noting the obvious, both to job-candidates but also--much more importantly--to graduate programs: namely, that it is better for job-candidates not to have to struggle with this dilemma at all! In my experience, although some graduate programs treat effective teaching as an important part of graduate education, others [or so I have heard] appear to focus primarily on research, treating teaching experience and effectiveness as something of an afterthought. In today's academic job-market, where jobs are scarce--and in which different types of jobs [teaching and research jobs] can have very different focuses/priorities--it seems increasingly important to have a strong research and teaching record, something graduate programs and job-candidates should not forget!

Anyway, what do you all think? What is the best way to deal with the kind of dilemma above?

Comments

[GF-A]

I have been on hiring committees at a SLAC several times. This post mostly matches my experiences. Just to add one wrinkle: my sense is that only including some of your classes' evaluations is not a huge red flag, UNLESS the only evaluations included in the dossier are from your small, upper-division classes. I think if you've taught 12 classes, leaving some evaluations out is not a red flag at all -- you don't need a 250-page dossier, and many search committees won't want one that long.

So: if you have seriously up-and-down evaluations, I would recommend including your best intro-level one(s), even if it's not fantastic, and maybe a selection of other ones where you have been in more 'challenging' teaching circumstances (e.g. bigger classes). The search committee sees their own evaluations every term; they know that, in general, teachers get lower scores on intro-level classes. And the committee is almost certainly looking for someone who will teach some intro-level classes.

[M.B.W.]

One problem with full disclosure is that for someone with a lot of teaching experience, the sheer volume of responses threatens to drown out any useful information.

Moreover, every school requires different things from evals. A place I worked required a student comment for every numerical rating! I cherry picked just so I wouldn't have a 100 page dossier. You see the problem: once I'm editing it, or would be presumed to be, why would I leave the worst comments in? Might not the committee assume the worst are my best?

I also think it's potentially unfair for members of groups who tend to be judged more harshly to abide by a norm of full disclosure. I don't think I need to make my first impression to colleagues out of crude comments on my appearance.

As a search committee member at a teaching school, I place less weight on comments and more on getting a sense of what the evaluation numbers mean. If you can compare yourself to departmental averages, that helps give context to the numbers. Even better, get someone to observe your teaching - have them write a letter.

[Derek Bowman]

The length issue that GF-A and M.B.W. mention is the reason I changed to only including selected evaluations. If I included all the student evaluations just for my first year, it would have been 2 pages times 140 students (35 per class, two classes per term) = 240 pages. I can (and have) compile and edit these down so they fit on fewer pages. But then it's still dozens of very dense pages. What committee - even at a teaching school - is going to comb through all of those?

From the applicant side it seemed like the choice was between a complete and representative picture of my teaching that no one would read or an incomplete cherry-picked version that has a small chance of catching someone's eye, at which point they can ask for more. I also think the comments I've cherry picked are informative, praising specific aspects of my teaching practice, rather than just generic positives.

[Derek Bowman]

Math fail. 280.

[Marcus Arvan]

Hi Derek: There's another way around the length issue that doesn't make it potentially look like one is trying to hide negative comments. Simply include complete and unedited comments from one term only. I have about 140 students per year [which is your one term average], and I include all student comments. It does not make my portfolio too long.

[Marcus Arvan]

Hi M.B.W: I'm not sure about the length issue. I didn't mean to imply that one should include all students ever. Rather, my suggestion is that one should include all student comments from several courses OR academic years. I include full and unedited student comments from a couple of semesters, and it doesn't make my portfolio too long. Admittedly, I teach relatively small classes [25 students per class, 75 per semester], but I include several semesters worth of comments, and my portfolio is a very manageable length. If I were teaching larger classes, I would probably just include one semester of complete/unedited comments, and I think that would still be manageable.

I think your point about groups who tend to be judged more harshly is much more concerning--and I certainly understand not wanting to include [e.g.] crude comments on one's appearance. I cannot help but think that this is something that at least in part should be addressed at universities in the student evaluation process. No one should have to endure crude comments about their appearance--and indeed, such comments seem to me a hostile work environment. However, I realize that even if such comments were eliminated, this would not resolve the general problem you are pointing to. If some groups tend to receive more negative and/or meanspirited comments in evaluations, full disclosure may well put members of such groups at an unjust disadvantage.

[Derek Bowman]

How many pages is your teaching portfolio, then, Marcus? For one of my semesters with 70 students the consolidated evals alone are 7-8 fairly dense pages. (I'm not sure what it would look like with my current 4/4 load with different class sizes and differently structured evals).

On the hiring side, how many applications did you carefully read 8+ pages of evaluations for?

[Marcus Arvan]

Hi Derek: I had a couple of versions of my teaching portfolio.

The version I sent to research schools was 31 pages in total, with one full semester's worth of single-spaced consolidated student comments--which was 5 pages.

The version I sent to teaching schools was much longer--about 50-60 pages, with several semesters' worth of comments.

I cannot say how many applications I read 8+ pages of evaluations for. But I think your question there might sort of miss the forest for trees. The most salient question is not whether people actually read every page of every application [as different people plausibly look for different things in an application, and some early "cuts" might be based on CVs, writing samples, etc.]. Rather, the most salient question is whether reading the phrase "selected student comments" might raise a yellow/red flag in readers' minds, leaving them to wonder what sorts of comments might have been left out. Further, I also suspect that when it comes to final decisions about who to interview/invite to campus in particular--both of which are obviously very important stages of the selection process--the proportion of files that get read very carefully, page by page, probably increases dramatically.

[Jerry Green]

#dang. And I thought my teaching portfolio was long (c. 30 pp)!.

Just to add to Marcus's 9:15 comment. I did something similar: one unedited set of comments (c. 12 out of a 20-25 person class), and linked to my website where I posted the rest.

I think this approach strikes the best balance you can get away with. On the one hand, since you've got a full set of unedited comments, and make the rest available, you're not trying to hide anything, and I think there's something to be said for a confident display of your information. On the other, the only people who care enough to actually go luck at the info online will know enough about teaching to understand how variable written comments can be, and how common it is to get that one very negative outlier.

[Derek Bowman]

Thanks, Marcus. I'm sure you're right re: forest/trees. Yet another case where it's a mistake to think of the information presented in a job dossier as a genuine attempt at direct communication by way of the information presented.