What is evidence of teaching effectiveness?

In our most recent 'Ask a Search-Committee Member' thread, Throwaway Name writes:

Could search committee members discuss their thinking when they evaluate, on paper, whether a candidate is a good teacher? Do they weigh certain application materials more than others (e.g. teaching statement, letters of recommendation, student evaluations, etc.)? In their own experience, are strong application materials of a certain kind correlated to strong teaching abilities? Are strong application materials of another kind not correlated to strong teaching abilities?

Anon added:

I very much second "Throwaway Name's" post. "Evidence of Teaching Effectiveness" is often a standard phrase for jobs, but how is that evaluated? How does that stand out? I really think this deserves a thread of its own.

I think this is a really good query, as I think how search committees think about the issue may be changing a bit. Allow me to explain.

My general sense is that, until recently, "evidence of teaching effectiveness" was sort of a code-phrase for student evaluations. However, my sense now is that many people are increasingly skeptical of the value of student evaluations. Should they be? The empirical picture is unclear to me. On the one hand, there is evidence student evaluation of instructor performance are affected my performance-irrelevant biases, such as "hotness." On the other hand, some studies (and meta-analyses) suggest that evaluations are positively related to student performance.

In any case, I expect many search committee members pay at least some attention to student evaluations--not just scores, but also student comments (and, as I've said before, I think it's important to include all student comments for a course, not just "selected" comments). However, I expect that "evidence of teaching effectiveness" is probably understood by many as meaning more than this. What else might count?

Good syllabi, I think, can be one piece of evidence of teaching-effectiveness. Suppose one candidate has rather minimal and unimaginative syllabi, whereas another candidate's syllabi are really thoughtful, well-developed, and more imaginative (in terms of reading-selections, types of assignments, etc.). People on hiring committees may have some idea of which sorts of teaching strategies work well at their school, given their student population, class-sizes, class-lengths, etc.

Creative, pedagogically well-justified assignments may be another source of evidence of teaching effectiveness. Consider argument-mapping. Suppose one candidate reports that their teaching methods just include lecturing, tests, and term-papers but another uses argument-mapping, assignments explicitly designed to develop student meta-cognition, final presentations that apply relevant course concepts to the modern world, etc. The latter person can make a plausible claim to developing more skills in students than the former--as their teaching methods are designed to get students to do things the other person's style does not.

Finally, a strong teaching-statement may in a way be considered "evidence of teaching effectiveness." If the statement makes a persuasive case to the reader how your teaching style is systematically designed to improve student learning (in different ways), then that--along with a holistic evaluation of other teaching materials--may be considered some "evidence of teaching effectiveness."

But these are just a few of my thoughts? What are yours, particularly those of you who have served on search committees?

Comments

[Paul]

I think that these are all good indicators Marcus. I just evaluated grad student teaching awards, and another factor is pedagogical development and (possibly) teaching awards. Obviously the latter are hard to come by, but if you come from a major institution I imagine you have resources available to you. Here we have an academy for teaching and learning, which puts on everything from lectures on specific ideas to improve teaching, to round table discussions, to syllabi revamping bootcamps, summer institutes and more. If a candidate had solid evaluations, thoughtful syllabi and teaching statement, and teaching development courses or activities, that I think would really set you apart. Do these extracurricular activities guarantee effectiveness? Of course not, but it does show the committee that you care and that you are taking concrete steps to improve your teaching.

[Michael Brent]

I think that, for the purposes of wooing hiring committees, a "good syllabi" will be good when it contains the sort of creative, pedagogically well-justified assignments that you mention, which are described and justified in your statement of pedagogy. Other than a good syllabus or two, your statement of pedagogy, and teaching evaluation data, indications of teaching effectiveness can include (1) a letter from someone who has observed your teaching, (2) a letter from a student with whom you have worked, (3) attendance at various pedagogy-focused workshops such as those offered by the Center for Teaching and Learning (or equivalent) at your institution, or the APA Teaching meetings, (4) awards and other formal recognitions, (5) written or published engagement with the scholarship of teaching and learning, and more!

[Asst Prof]

I have never served on a hiring committee, but my thought is that part of a teaching portfolio can serve as evidence for the claims in a teaching statement. For example, if my teaching statement follows Marcus’s past advice and describes concrete assignments and activities I have used to engage students, I can then also provide the full text of these assignments and activity prompts in a teaching portfolio for hiring committee members who want to peruse them in more detail.

Even things like the graphic design of assignments (is there adequate white space, clear headings) and the way things are phrased (are assignments encouraging, open-minded, clear, tied to learning outcomes) can potentially reveal one’s practices and character as a teacher.