How to know if you're improving at online teaching? (Guest post by Wes Siscoe)

Wes Siscoe is a Dean’s Postdoctoral Fellow at Florida State University and the Mellon Course Design Coordinator for the Philosophy as a Way of Life Project at the University of Notre Dame. He received his PhD from the University of Arizona and has been a visiting researcher at Brown, Notre Dame, and Rutgers.  His research revolves around several themes – rationality, language, and virtue – and their importance for accounts of human excellence and achievement.

How Can I Tell if I am Improving at Online Teaching?

It is often difficult to tell when you're improving as an instructor.  Good student evaluations can be manufactured by watering down the course content, and grading can be inconsistent and inflated from semester to semester. These problems are exacerbated by the online environment - with less face-to-face contact with students, it is even more difficult to tell how student outcomes compare to past semesters. In this post, I will provide a step-by-step outline of our research-driven methodology for assessing the courses we have created at the Philosophy as a Way of Life Project, an approach which remains effective even when the majority of instruction happens in the digital classroom.

Step 1: Student Evaluations for High Expectations

The jury is still out on whether student evaluations are strong indicators of student learning. While some have argued that student course evaluations are good measures of teaching effectiveness, others argue that there is no correlation at all between excellent student evaluations and great student outcomes. Students are notoriously swayed by lenient grading and other factors that are irrelevant to effective instruction, and they are often not equipped to judge their own competence when learning novel content. So is there any reason at all to look to student evaluations when attempting to improve as a professor?

Even though it is unclear whether student evaluations are useful indicators of overall course quality, they do represent one thing very well – how high the expectations are for students. Students are quick to report a course’s workload as demanding or challenging, especially when the course is more difficult than they expected it to be. Course evaluations can be useful, then, for showing how rigorous students perceive a course to be, even when that perception does not correlate with student satisfaction. If students report high satisfaction ratings but little to no work, it might be time to make the course a bit more difficult.  On the other hand, if students are worked to the bone, the next step might be easing back the course requirements so that students can enjoy philosophy instead of just suffering through it.

Step 2: Timely Feedback for Academic Support

Creating a rigorous course will prompt students to rise to the challenge, and giving timely feedback is what will make it possible for them to meet your expectations. Having high expectations for students is good, so long as students have the necessary support to succeed once a course becomes challenging.  Though more research is needed to settle the size of the effect, building a course around giving efficient, detailed feedback has been shown to be a boon for student learning. 

Strategies for providing additional feedback to philosophy courses include incorporating instantly graded multiple choice quizzes, peer-reviewed short answer questions, and essay tutorial meetings. All of these strategies provide students with feedback that allows them to improve on past performance, learning from past mistakes how they can do better work moving forward.  Whatever of feedback you incorporate, make a plan for how you will give that feedback quickly and efficiently. A long delay in returning student work prevents them from using your comments as effective feedback, whereas shortening your feedback loop allows students to use the evaluation they receive to make further improvements.

Step 3: Common Assessment Items for Overall Improvement

Once you have a rigorous course that gives timely feedback, you will need to measure that class against previous student performances to see how much these changes have improved on semesters past. We have already established that grades might not do the trick.  If past classes were less challenging or had any grade inflation, then overall course grades might actually go down.  The best strategy for judging whether students are actually improving is by including common assessment items that you use both in your past and present course. Common assessment items can include identical multiple choice or short answer questions, or even similar essay prompts. These common assignments will allow you to directly compare the past and present courses to see if student performances are making progress overall.

The primary advantage to using common test items to judge improvement in a course is their objectivity. We have discussed the complications with the other markers of course improvement – course evaluations and grades are too noisy to draw any hard conclusions about whether you have made progress in teaching a course.  Common assessment items are independent enough from these confounding factors that they can provide a measure of objectivity to course comparisons. Now, just one or two common assessment items might not be enough.  You do not need to replicate all of the assignments from a past course, but you should have similar items across a number of assignments in order to get a full picture of whether students are improving on past class sections. After ensuring that your course is challenging using student evaluations and providing enough support through constant, efficient feedback, common assessment items will enable you to judge whether these adjustments have led to lasting course growth.

I have summarized a general strategy for evaluating and improving your online instruction, a schema that you can tweak or adjust as your classroom demands. For instance, it may seem strange that we have had an entire conversation about improved student outcomes without mentioning grades – after all, grades are often used as the primary measure of whether students pass or fail a class. Because of the number of complicating factors with inconsistent assessment and grade inflation, however, an improvement in grades does not mean much unless it is paired with all of the previous steps. If grades are not the primary focus, however, you can feel free to use them in other ways. Perhaps short, daily comprehension quizzes keeps students honest with the scheduled readings. Maybe giving more participation grades gets more students talking, creating a better sense of community. If your class is challenging and students are improving via frequent feedback, the good grades will come, regardless of whether you are teaching in person or online.

Comments

[Evan]

Disagreements about whether or not students have learned something often stem from people’s various differences in expectations of what they consider or conceptualize the phenomenon ”learned” to be. This word is past-tensed because that’s what we’re concerned about. At the end of the day, it’s what most teachers wanna know: *did* their students learn?

The answer to this question really depends on our conceptualization of ”learn.” The word ”learn” implies having known something or at least, memorized it enough to recall it upon exam or quiz time. If that’s your idea of what it means to learn something, then there are ways to measure it such as taking exams and quizzes. This is commonsense.

I like to think that education should aim at *more* than just figuring out whether or not students have learned something. Instead, we should also ask if they actually *understand* something they’re being taught. Understanding involves more than just rote memorization. Rather, it involves the *application* of said knowledge.

A word of caution: don’t assume your students already know-how to do something. If you want your students to interpret a text, you must give them the principles or tools of interpretation. I remember I was told to interpret a text once for an assignment. I did and then I got a bad grade on it because my teacher was dissatisfied with my interpretation. Granted, I was ignorant of the critical thinking tools at the time. My educational experience has been such a haze looking back. I did not know that there were various ways we can and should interpret a text.

Don’t do this to your students unless they’re not being graded on it. It’s unfair to them and can be harmful to their self-esteem and intellectual development. Instead, do what math and art teachers do: demonstrate first and then let them apply afterward. After all, demonstration is a sub-species of instruction (species of teaching). Teaching involves a kind of show-and-tell.

[Wes Siscoe]

Thanks for the input, Evan! I think that you’re right that, when it comes to philosophy classrooms, students often do not receive enough modeling as part of the instructional experience. Even if it seems like common sense to someone who has been reading philosophy for years, it is not common sense how to read or write philosophy, and that necessitates showing students how it’s done. The most effective instructors are often those who can empathize when their students, accurately understand the level that they are at, and then start modeling what it would look like for them to move forward in terms of reading and understanding philosophy. Thanks for the great contribution!

[Evan]

Wes: You’re welcome!

From personal experience, I know there is generally a big difference between teaching at a poor inner-city school and a prestigious school. I went through inner-city schools most of my life with many students who were Black and Latina/o.

My cousin, however, lived in Boston, which was 45 minutes away from where I lived. She went to Boston Latin Academy which was ranked second place (at the time) of Boston’s best high schools. Students have to take an exam to even get into it. Students there had to study Latin while reading books like The Aeneid by Virgil. They were taught how to translate and interpret Latin texts. The students there were more ”cultured” I suppose. The biggest difference between my school and hers was that our school was given more to memorize while she and her classmates were given less to memorize and more to understand. This is how it is in rich schools so we shouldn’t be surprised.

I’m not the most smartest haha. But I’ve always been a great and clear writer. That much I’m proud of at least.