Testing What We Mean to Test
If the courses you teach include exams, you may be working on preparing the first one around this time of the semester. Many faculty use exams to evaluate how much students have learned at particular points in a course, but designing an exam that does so effectively can be easier said than done.
When you give an exam, you’re gathering data on your students’ progress, so you want it to be both valid and reliable. First, as Ory and Ryan put it, you have to make sure you’re “testing what you want to be testing”—that is, what your students should know and be able to do. If your test is not aligned with your learning goals for students, “you will not be able to evaluate your students’ progress with any accuracy,” explain Clegg and Cashin. This means you need for the tasks on the exam—the questions, problems, etc.—to elicit the kind of thinking that you’re trying to test.
For example, you’ll need to consider what proportion of each exam will test higher-order thinking as opposed to recall. Many test banks tend to focus on recognition questions rather than analysis. If you want your students to be able to apply concepts in a variety of ways, you’ll need questions that demand that sort of thinking. Of course, your students will also need to practice this thinking in class, repeatedly. We can’t expect them to pick up new reasoning skills during an exam.
It’s useful to think about how much time and priority you allot to particular concepts in the time leading up to the exam. The most important ideas deserve the bulk of our attention, student effort, and class time, so they merit a corresponding proportion of exams. As Mastascusa et al. point out, research on the “testing effect” shows that if we test primarily on minutiae, our students may leave the course remembering those details, rather than the foundational concepts.
Tests and quizzes don’t just measure performance; they can be powerful learning opportunities. We’re asking students to retrieve and apply the ideas they need to take away from our courses, so ideally we’re leading them through the thinking we want them to practice. Students can also learn from exams by completing exam wrappers. These activities help students reflect on their preparation and performance on an exam, identify gaps in their current knowledge and skills, and strategize how they can fill those gaps and improve their performance on the next exam.
If you’re working on an exam right now, practical guidelines and suggestions plus a checklist for effective tests are available here. You can also explore this more in-depth guide to writing good multiple-choice questions. CMU offers broadly applicable tips on designing effective exams. And as always, we’re available to help: feel free to bring us your draft exams if you want feedback! You can reach out for a consultation by emailing us at pro-teaching@fsu.edu. We look forward to working with you!