Making the Most of Multiple-Choice Questions : Getting Beyond Remembering

119 M testing is one of the most commonly used forms of assessment in college and university settings. Compared to constructedresponse test questions, such as essays, multiplechoice questions enjoy several important practical advantages. For instance, for a test of a given length, well-chosen multiple-choice questions can provide a broader coverage of course content than constructedresponse questions, and moreover, their scoring is generally more statistically reliable. In addition, with large classes, the time and effort required to grade students’ written answers can be enormous. In contrast, the grading of multiple-choice tests is far easier, especially when students have used a response sheet that is scored using an optical scanner. Despite these substantial advantages, many teachers tend to have quite a low opinion of multiple-choice testing. One major reason for this was brought to me in full force several years ago during a discussion I had with Jenny, an award-winning science teacher. As so often happens when two teachers get together, the conversation soon turned to workrelated matters – favourite topics to teach, how to get students to read the assigned readings, strategies for generating discussion in the classroom. At one point in the conversation, I mentioned my then new-found interest in multiple-choice testing and was about to tell Jenny about a research project that I was just getting off the ground. However, before I could even get started, she cut me off at the knees. “Multiple21

M ultiple-choice testing is one of the most com- monly used forms of assessment in college and university settings.Compared to constructedresponse test questions, such as essays, multiplechoice questions enjoy several important practical advantages.For instance, for a test of a given length, well-chosen multiple-choice questions can provide a broader coverage of course content than constructedresponse questions, and moreover, their scoring is generally more statistically reliable.In addition, with large classes, the time and effort required to grade students' written answers can be enormous.In contrast, the grading of multiple-choice tests is far easier, especially when students have used a response sheet that is scored using an optical scanner.
Despite these substantial advantages, many teachers tend to have quite a low opinion of multiple-choice testing.One major reason for this was brought to me in full force several years ago during a discussion I had with Jenny, an award-winning science teacher.As so often happens when two teachers get together, the conversation soon turned to workrelated matters -favourite topics to teach, how to get students to read the assigned readings, strategies for generating discussion in the classroom.At one point in the conversation, I mentioned my then new-found interest in multiple-choice testing and was about to tell Jenny about a research project that I was just getting off the ground.However, before I could even get started, she cut me off at the knees."Multiple-Introduction choice?Are you serious?I would never dream of using multiple-choice on my tests!All they are good for is finding out how well students can memorize, and who cares about that?"At the time, I was quite surprised by Jenny's strong reaction to my mention of multiple-choice testing, but since then I have come to realize that a great many teachers are much like Jenny -that is, they believe that multiple-choice tests are only good for assessing students' ability to memorize, and that they are not useful for assessing higher-order cognitive skills.Research that my students and I have recently carried out at Brock University suggests that this perception is actually quite widespread.In a broad survey of teachers at Brock, we found that about 90% believe that multiple-choice questions are an effective way to assess students' ability to remember information.However, far fewer of them believe that multiple-choice questions can effectively assess students' ability to understand (46%), apply (23%), analyze (59%), and evaluate (16%) information (Anderson & Krathwohl, 2001).Thus, the belief seems to be quite common among university teachers that multiple-choice questions are primarily useful for assessing students' ability to memorize, but not much else.
Although I would certainly agree that multiple-choice questions are an extremely good way to test students' memory, I would also argue that they can do much more than this.Like a number of others who have thoughtfully considered this issue (Haladyna, 1999;Linn & Gronlund, 1995), I believe that multiple-choice questions can indeed be used to assess the higher-level cognitive processes mentioned above -understanding, application, analysis, and evaluation.For the past several years, I have been presenting workshops at colleges and universities across Canada in which I familiarize teachers with strategies for generating multiplechoice questions that will do more than test students' ability to remember information.One such strategy involves taking an existing multiple-choice question that tests memory and "ramping it up" to a higher cognitive level.I will provide here just one example of how this might be done.Consider the following question that might be found in the mul- This question asks students to do nothing more than remember a fact -namely, that an unconditioned stimulus elicits a particular response without any prior training.Note that to answer this question correctly, students do not have to know anything at all about the details of classical conditioning or how it works; by doing nothing more than remembering the definition for unconditioned stimulus, they can answer Question 1 correctly.Indeed, students who cram the night before the test might very well memorize the definition for unconditioned stimulus, answer Question 1 correctly, and then promptly forget the information immediately afterward.
Unfortunately, low-level items like Question 1 are often encountered in test banks, but the good news is that we can use this item as a starting point to generate a question that requires students to have learned something important about classical conditioning.So how might we ramp this "remember" question up to a higher cognitive level?One way would be to require students to demonstrate some understanding of what an unconditioned stimulus is in the context of a situation that they have not previously encountered, like this: Question 1-Revised.Right after a rat smells menthol, it is always given Drug X, which reliably induces substantial water intake.Eventually, the rat drinks water whenever it smells menthol, even when it is not injected with Drug X.In this situation, what is the role of Drug X? Notice that the four alternatives that are provided have not changed at all, and the correct answer is still alternative B. However, Question 1-Revised requires students to have an understanding of the concepts underlying classical conditioning, and furthermore to be able to classify the role of Drug X in this research setting that they are seeing for the first time.With Question 1-Revised, students who have simply memorized the definition for unconditioned stimulus will be at a loss, while students who have truly learned about classical conditioning and understand how it works will likely be able to answer this question correctly.
It is also possible to generate questions that assess higher-level cognitive processes by requiring students to read material as part of their course work and then asking multiple-choice questions that depend on their having determined the author's point of view, bias, values, or intent.To illustrate, let's consider a question that I have recently used in a psychology course that I teach.In the course, students read How to Think Straight About Psychology, an outstanding book by Keith Stanovich (2007) that provides a thoughtful examination of critical issues in the sciences generally and in psychology in particular.I provide students with study questions that serve as a guide as they do their reading, and in class we touch upon many of the issues that are dealt with in the book.Question 2 is typical of the type of question that I would ask students on a test.Note that, as in the previous example, students have not encountered the particular situation that is presented in the question.To answer this question, students must have read and understood Stanovich's discussion of what he calls "person-who" statistics, which involve the (mis)use of data from a single case to argue against some wellfounded statistical conclusion ("How can you say the odds of winning are against you at the casino?I know a person who went to the casino and won $10,000!").Note also that Question 2 does not simply ask students whether they can recognize the definition of person-who statistics, but rather it requires them to understand the underlying concept.Moreover, it calls on their analytical skills by requiring them to attribute to the author a particular point of view and to apply that point of view to a situation that they have not previously encountered.Thus, Question 2 goes far beyond simply testing students' ability to remember information that they have encountered in the course.
A third strategy that I encourage teachers to consider involves the use of interpretive exercises, which are sometimes referred to as item sets (Haladyna, 1992).In an interpretive exercise, students are first presented with novel introductory material, such as a reading, scenario, data set, chart, or map.They must then use the skills that they have developed in the course to answer a series of multiple-choice questions that relate to the introductory material.The following is the introductory information from one of the interpretive exercises that I have used when teaching introductory statistics: Madame Clousseau claims to be a psychic -that is, she claims to be able to predict future events with a level of accuracy that is substantially better than chance.To examine her claim, Professor Jones brings her into his laboratory and tests her under carefully controlled conditions.He tosses a standard, fair coin 300 times and has Madame Clousseau predict what the outcome will be for each toss.He finds that she correctly predicts the outcome for 157 of the tosses.When he carries out the statistical test to analyze the results, Professor Jones lets alpha equal 0.05 and he uses a twotailed test.
This research scenario, which of course the students have never seen before the test, is followed by a series of four-alternative multiple-choice questions, such as "Which statistical test should Professor Jones use to analyze the data?" and "What would be the critical value(s) for the test statistic?"Note that students can correctly answer these multiple-choice questions only by referring to the introductory information.Moreover, answering correctly is not simply an exercise in fact-finding.Rather, students must be able to understand and interpret what they have read, and they must then call on the statistical skills that they have developed in the course to answer the questions correctly.Thus, this interpretive exercise goes well beyond testing whether students remember facts and concepts that they have encountered in the course, and instead calls on a variety of higher-level cognitive processes.
Generally speaking, interpretive exercises are quite robust because they can be used for testing in a wide variety of disciplines.For instance, a teacher of poetry might have students read a poem for the first time and then answer multiple-choice questions dealing with its form, style, and content.Or a music teacher might have students listen to a recording for the first time and answer multiple-choice questions that involve the identification of vocal and instrumental components as well as elements of form.
I have presented several strategies that can help teachers to generate multiple-choice questions that assess higher-level cognitive processes.I am fully aware, of course, that multiple-choice questions have clear limitations.For instance, they are not at all useful for determining whether students can structure an argument, develop novel ideas, or invent something new.Such abilities are best assessed using other techniques, such as essays, term papers, and projects.Nevertheless, multiple-choice questions can be used on tests to assess higher-level cognitive processes in ways that many teachers seem to overlook.Moreover, some teachers who have underestimated the value of multiple-choice questions may find that they can play quite a useful role in the assessment process.This is especially true for larger classes in which the grading of written answers can involve considerable time and effort.
tiple-choice test bank accompanying an introductory psychology textbook: Question 1.In classical conditioning, what name is give to a stimulus that elicits a particular response even in the absence of any prior training?

Question 2 .
According to census data, people are having fewer children nowadays than they did 50 years ago.Your friend Anne tells you that she does not believe this because the young couple who live next door to her are both under 30 and already have four children.If Keith Stanovich were told about this, what might you reasonably expect him to say? A. The census data must be wrong.B. Anne's comment illustrates valid probabilistic reasoning.C. Anne's comment illustrates the use of "person-who" statistics.*D. The young couple provide an exception that actually serves to prove the rule.