How Are We Measuring Learning?
Finals week is sneaking up on us this spring. This semester has been another tough one: students are exhausted and isolated, as well as frightened and distressed by the multiple acts of violence in the news. (Many faculty and TAs are similarly overwhelmed, of course). This is an important time to be on the lookout for students in distress, and to share resources like those we compiled last fall. At the same time, we’re making plans to gauge how much our students have learned. We begin every semester–even a pandemic semester–with high aspirations for how our courses will help our students to develop intellectually, professionally, and personally. While we hope we have created effective remote, hybrid, and flex learning experiences, when we account for the stress we’ve all (students and faculty alike) been under, we have to be both humane and realistic when we assess the progress our students have made this year.
The purpose of any assessment is to collect evidence of students’ progress toward the learning goals in the course. Exams are traditional tools for evaluating learning, but they’re not always the most effective means of gathering this data. There are plenty of alternatives; but if you’re giving exams, they need to be both valid and reliable measures, so they may not look like the exams we remember taking as students. When we’re “testing what we want to be testing,” as Ory and Ryan (1993) put it, we start with a clear idea of what successful learning would look like: what should students know or be able to do when they finish the course? Then we figure out how students can best give us evidence of that learning.
To devise a fair and well-aligned exam, you’ll have to consider how much time and priority you allotted to particular concepts this semester. The most important ideas deserve the bulk of attention, student effort and class time, so they merit a corresponding proportion of exams. As the research on the “testing effect” reminds us, if we test primarily on minutiae, our students may leave the course remembering those details, rather than the foundational concepts (Mastascusa, 2011). Tests and quizzes don’t just measure learning; they should also be powerful learning opportunities. We’re asking students to retrieve and apply the ideas they need to take away from our courses, so ideally our exams lead students through the thinking we want them to practice habitually. The final is our last chance to direct their attention.
To make sure your exam is balanced, and reflects your priorities for learning, you might want to draw up a Table of Specifications, where you identify the major concepts and their weight, assigning each concept the number and difficulty of questions that will adequately measure students’ level of mastery. If you want your students to be able to apply concepts in a variety of ways, you’ll need questions that demand that sort of thinking rather than just calling for recognition. Many test banks tend to rely on recall questions, rather than analysis, so you will probably need to develop your own–and we’re happy to help you write great questions! Of course, your students will also need multiple opportunities to practice this thinking beforehand, in class and in homework, or lab, or recitation, etc.–we can’t expect them to pick up new reasoning skills the day of the exam.
Practice tests are a great idea, especially if you have old exams you can use. Students will feel more confident if they know what sorts of questions to expect, and how many of various types (essays, problems to solve, MCQs, etc.) Especially if they’re taking exams online and can’t scan over the whole test and then return to work the challenging problems, students may have difficulty pacing themselves, so a trial run will help assuage their anxieties.
For additional practical guidelines and a checklist for creating effective tests, you can click here; and here’s a great handout on designing effective exams from CMU. And as always, we’re available to help: feel free to bring us your draft exams, or talk over your plans.