Teaching in the era of ChatGPT?

Like many of us, I saw the news the other day that the newest natural language AI models, such as ChatGPT, are increasingly able to create grammatical and sophisticated responses to a wide variety of prompts. And, in fact, the other night a family member showed me an example in real time: a believable four-paragraph essay composed in just seconds.

Now, of course, there is already debate about whether this means that "the college essay is dead", or whether AI still can't compose sufficiently good content to enable rampant academic cheating (at least not for good grades). But, I guess that, given what I've seen so far, I'm leaning toward the former camp. These are still relatively early days, and the pace at which AI is improving is astonishing. 

So, what does this mean for us as instructors? I have to confess that I've already given some thought to giving my pedagogy a vast overhaul, perhaps moving away from at-home assignments to handwritten in-class ones. But, for all that, I'm deeply ensure what to do, or whether to pursue any dramatic changes at all. Are any readers giving thought to these matters? If so, what do you plan to do and why? Any clever ideas for new pedagogies to perhaps experiment with?

I'm also curious what these developments mean for university-level general education curricula. At many universities, general education programs increasingly focus on "high impact practices" such as writing and presentations. Will universities need to radically rethink their general education curricula and learning outcomes?

Comments

[Michel]

I have always thought that, for lower-level classes, in-class essays are a better assessment tool than take-home essays. They remain unaffected by LLMs.

I'm shifting back to them, myself. They also cut down on ordinary plagiarism, which is nice.

[anon]

(i) I might end up doing more in-person assessments.

(ii) I have done some experimenting which has made be a bit less worried, but I don't know how representative/reliable my results are, so would be curious to hear what folks think about the following.

It's seemed to me that with very short prompts ("Tell me why you don't like Kantian ethics, 500 words") it generates fairly different essays from attempt to attempt. The prompts I use are longer and more detailed - this semester I have a roughly 600 word prompt for a 1000 word essay, for instance.

When I fed this prompt into GPTChat on multiple occasions, the result was substantially the same each time - a few sentences here or there are different, occasionally a new point of argument, and so on. So maybe I have reason to be optimistic that anti-plagiarism software which collects the student papers and compares them against each other would catch this, as long as more than one student tries to do it. (My classes are often 200+ and graded by TAs.)

But I haven't tested this extensively, maybe I'm way off.

And even if this is a real effect, maybe it will go away eventually. Or if it's a real effect, students could catch on to the risk and do some individualizing.

[SLAC assistant]

I'm torn. On the one hand, I know the limited pedagogical value of tests and other high-stakes, in-class forms of assessment. On the other hand, it's becoming harder and harder to think of an alternative. I've been experimenting the last year or so with less traditional assignments--memes, haikus, etc.--but even these can now be gamed by AI. I do think that whatever we do, we shouldn't make too much our jobs into an attempt to block cheating. It's going to happen to an extent regardless--no system is perfectly efficient--and at the end of the day we need to teach to those who care enough about their education to do the work themselves.

[grymes]

1) People have already developed ways to detect AI-generated text: https://huggingface.co/openai-detector There will be an arms race here, but I see no reason to think the cheaters will significantly outpace the detectors.

2) As for rethinking pedagogy, I'll just note that cheater-detection is not a part of pedagogy proper. Neither is changing assignments for the express purpose of making it harder for students to cheat. Those activities are part of another activity--administration carried out by employees of degree-granting institutions--which teachers are contractually obligated to balance with pedagogy. I'm not saying that we shouldn't fulfill this obligation--or even that it's impossible that changing assignments for administrative reasons might have good pedagogical consequences. But I do think it's important to keep these two aspects of our job clearly distinct in our minds when deciding what we're willing to sacrifice in order to better detect cheating.

[William Peden]

Given grade inflation and the low ambitions of many students, these already pose a huge problem. Instructors in many universities are under pressure to pass anything that gives the impression of a student making some effort.

I don't know the precise motivations of oral examination in the past, but they were once a big thing in university education. The challenge would be that a shift to more oral examination would require a lot more instructors, but from my perspective as a young academic, that would not be a bad thing!

I would also like a shift to classroom examinations as a method of continuous assessment, since they test both a student's conscientiousness and their ability to work under pressure, which are extremely useful skills in life. Insofar as the essay questions are tests of understanding rather than knowledge, I wouldn't even oppose students taking in e.g. a paper notebook of info that they want to use.

[grymes]

(I didn’t see SLAC Assistant’s excellent comment before leaving my own. If I had, I wouldn’t have felt the need to make point 2.)

[Craig]

I just had a fantastic conversation with ChatGPT. I asked it to do something dumb (write joke wedding vows), and it refused, saying that doing so would be inappropriate. We then had a little back and forth about whether the ChatGPT has ethical beliefs, about where it gets them from, about how it converts those ethical beliefs into action. I'm not teaching a meta-ethics course this semester

[The rest of Craig]

... but I could easily imagine using conversations with ChatGPT to teach just such a class.

[anon-4]

There's a recent article about the merits of oral examination in NYT: https://www.nytimes.com/2022/12/02/opinion/college-oral-exam.html?action=click&module=Well&pgtype=Homepage&section=Sunday%20Opinion

[Ideas Man]

Is it OK if this comment thread turns into a sort of pedagogy brainstorming session?

I don't like in-class exams, and I don't trust my intro-level students to do take-home exams without cheating. I typically assign essays as my major assessments for intro-level, non-logic classes. I'm a bit worried about continuing to do this because of (a) AI and (b) I think it might be unfair to the current crop of incoming students at my university. More than half seem to come in with almost no writing skills at all, and my class sizes are too large to teach writing effectively. I don't think my students are even required to satisfy the composition requirement before taking some of my 1000-level classes, and it shows.

What other kinds of assessments could I use? Oral exams sound great, but my sections are typically more than 40 students with no TA help. I'm worried this would be a massive undertaking.

Some ideas I have had, if anyone has thoughts:
-Formalized, team debates at the end of the term.
-Require students to engage someone outside of class in a philosophical conversation and write up how it went. Maybe they could even chat with GPT.
-Write and stage a short philosophical play with one or two partners.
-Record podcasts with partners.

If other people have ideas, I'd be eager to hear them.

[Marcus Arvan]

@Ideas Man: by all means, brainstorm away!

[grymes]

@Ideas Man

I do recorded conversation assignments instead of essays in my intro course (for the same reasons you give), and they work great. After some experimentation, I've settled on having them spend about 5 minutes explaining a philosophical problem or view to a friend, drawing on at least one reading, and then 5-10 minutes discussing it with the friend. They're no quicker to grade than freshman essays, but they are less soul-crushing.

[Alex Grzankowski]

I haven’t given this careful thought but I’d like to ask students to grade and comment on a GPT answer to a question. Seeing what’s good and what’s not so good about an answer would bring out a lot of the same skills and knowledge that I’m typically looking for.

[Marcus Arvan]

@Alex Grzankowski: That's a super interesting suggestion!

[Anon Postdoc]

In response to a few of the suggestions above I think there might be a slight mistake going on: the problem the GPT chat etc produce is content generated by AI, not only traditional essay content generated by AI. It seems to me that some people are assuming that these AI systems won't be able to, for example, write a play or a haiku or a podcast script, or some other non traditional (for philosophy examination) type of text. But even if they can't right now (which you would need to extensively test actually establish) they may be able to do so in the near future. The advantage of oral exams (or even just closed book essay exams) is that the students can't generate them with AI and then just present it as their own work.

If this really is a problem, we don't need to just move away from essays written at home, we need to move away from anything preproduced at all.

Also, please don't force your students to do some `creative' task like write a poem or a play. These might be fine as options for students who want to do them. But a lot of people, like myself, probably would have dropped out of philosophy straight away if I had been forced to write a poem. Teaching philosophy is about teaching philosophical reasoning skills, transmitting philosophical knowledge, and assessing the students ability to demonstrate what they have learned in straightforward ways. Being able to write a poem should never be a requirement since it has nothing to do with being able to do philosophy.

[SLAC assistant]

I know this is a tangent, but I'll respond to Anon Postdoc, since I think they were referring to my assigning of haikus in a philosophy class. I understand your hesitations, but a few points:

1. Nobody is required to write artistically good poetry to get full credit.

2. Haikus are not worth a lot, credit-wise. It's not as if my whole class is determined by philosophical poetry.

3. No graded component of a course speaks to every single pedagogical goal. So, while there's little in the way of philosophical argumentation in a haiku, there's also little philosophical argumentation in many of the things we grade (e.g., a great way to get participation credit for many professors is simply to ask relevant questions, but asking a question is not an argument).

4. Even a haiku can require that a student demonstrate philosophical understanding, which is surely a pedagogical aim. After all, if you don't understand, say, the principle of universal law, you likely can't create a haiku demonstrating that understanding. This is especially true when we consider one of the key components of understanding--extension of content into new contexts.

5. Like many short assignments, haikus require that a student choose their words carefully. They have only 17 syllables to demonstrate understanding. I have found that many students put more care into haikus than they put into their papers, since in a paper one can often write carelessly and still get a passing grade of some sort as long as they're sufficiently in the ball park.

6. This is a lower-level class. For every student who is turned off by philosophy by being forced to write a poem or two (again, this is a small fraction of the final grade), there are likely several more who might actually develop more interest in philosophy or who will be turned off if their grade is solely determined by papers and exams. I do not teach at an elite institution. Using playful assignments--and I do grade them seriously, based on a rubric--is often the difference between generating sufficient student interest and not. For example, students get a kick out of when I present awards for best haiku or best meme. Nobody is going to get into grad school on the basis of this, but it does seem to make class better for students who are unlikely to take a second philosophy class.

[traditionalist]

Anon PostDoc
Right on! I was sick when I heard colleagues were assigning really non-conventional assignments in classes. It really said to students: "You won't be going graduate school ... so some bozo assignment is fine for you". I even question whether my colleagues were qualified to evaluate such work. Too often faculty really undermine their own jobs ... and say: "hey look, philosophy is silly and worthless".

[elisa freschi]

Thank you for the comments, everyone! I have recently asked a similar question on twitter (comments here: https://twitter.com/elisa_freschi/status/1601752941832327168) and several colleagues and friends added further advice (mainly: building papers step-by-step in class and/or through personal interactions, so that a different end-result would stand out immediately). Personally, I am thinking of using peer-reviews and a two-steps submission (draft->peer review—>resubmission) even in larger classes. And I will warn students at the beginning that an AI-paper will not receive any mark.

Here, I especially liked the idea of recording a talk (student explaining a paper to a friend+Q&A). Thanks again for the mutual support!

[C. D. Brown]

How about something like this:

Ask students to write a traditional essay, then have a brief in-class oral interview with each student asking them to explain the content of the essay (without them looking at it). If they are completely unable to do so, they get no credit. If they can explain the essay (and show understanding), then the essay gets graded as normal.

I'm thinking this is better than moving to purely oral examinations Without written assignments, any students seeking to go to grad school will not acquire important philosophy-specific writing skills. Allowing students to write essays at home, instead of in class, allows them to take longer and edit the paper (though I admit that this will be rare, especially for lower-level courses), and will be much easier to grade, since handwritten essays are often written in indecipherable chicken scratch.

[Ideas Man]

Similar to C. D. Brown, I have considered making students write a very simple, one-page outline of their thoughts on a certain topic and then do an interview similar to the one Brown describes. I feel like that might be a good compromise.

I also disagree with Anon Postdoc's perspective, but I appreciate seeing different people's views on different types of assignments.