What’s all the clicking about? A study of Classroom Response System use at the University of Toronto

Classroom response systems (clickers) are used in many courses at the University of Toronto (U of T), primarily to introduce interactive pedagogy and to engage students in lecture courses. We examined the use of clickers in various courses at U of T and interviewed over 30 instructors about their use of clickers in classes with a total enrolment of over 5,000 students. Students in these classes were surveyed about their perception of the value of this technology. The objectives of our study were to evaluate the logistics of using clickers, the pedagogical value and associated teaching strategies, and students’ perception of its efficacy in their learning. We discuss some of the successes and failures of using clickers as a teaching and learning tool.

here are a variety of procedures for using clickers reported in the literature, but generally, the instructor presents students with a conceptual multiple-choice question (Reay et al., 2005) and allots a speciic amount of time for students to answer the question before closing the voting.Typically, a discussion follows, and on occasion one or more post-questions are posed to check if students understand the concepts (Beatty, 2004;Crouch & Mazur, 2001;Mazur, 1997;Rao & DiCarlo, 2000). he pedagogical method surrounding clicker use is described as a process that encourages and advocates "peer instruction" (Mazur, 1997), improves students' problem solving abilities and performance on traditional quizzes (Rao & DiCarlo, 2000;Ruhl, Hughes & Schloss, 1987), and improves student engagement and learning outcomes (Beatty, 2004;Brueckner & MacPherson, 2004;Crouch & Mazur, 2001).It is also said to improve interactive classroom discourse, increase students' active participation, and increase ownership of their learning (Beatty, 2004;Dufresne et al., 1996;Rao & DiCarlo, 2000), while it decreases student anxiety (Owens & Walden, 2001), lower level learning, and passive rote memorization of lecture material (Rao & DiCarlo, 2000).
here are a number of advantages and disadvantages of clicker use reported in the literature, but the most common reported advantage is that of increased student engagement, and the most common reported disadvantage is the administrative burden associated with the technology (for example, see Beatty, 2004;Burnstein & Lederman, 2003;Carnevale, 2005;Crouch & Mazur, 2001;Dufresne et al., 1996;Fies & Marshall, 2005;Mazur, 1997;and Rao & Di-Carlo, 2000).
A systematic review of diferent clicker models suggests that many of the commercially available clickers are very similar (Burnstein & Lederman, 2003).In 2006, the University of Toronto (U of T) adopted a single clicker vendor for its three campuses, encouraging all instructors to use the same system.1 his decision enables students to purchase one clicker for multiple classes, and it allows the university to ofer resources and training to faculty on one system.By the spring of 2008, over 60 instructors at U of T were using these clickers as part of their teaching, and the U of T bookstores were reporting sales of over 10,000 clickers per year.
We surveyed U of T faculty and students in order to determine the following: • What types of classes are using clickers most successfully?
• What are the best pedagogical practices for teaching with clickers?• What are the best logistical practices for the administration and use of clickers?• Do students believe clickers help them learn?

Faculty Survey
We conducted 32 structured interviews to survey faculty from various departments (e.g., departments in the Faculty of Arts and Science, Faculty of Medicine, School of Management).All of the interviewees had some experience with clickers.We asked several questions about the nature of the classes for which they used clickers, the logistics of their clicker use, their teaching styles, and their opinions about the advantages and disadvantages of teaching with clickers.We chose this form of descriptive survey research to develop an in-depth, quantitative description of instructors' use of clickers in the classroom.he resulting data and descriptive statistical analyses are presented below.

Logistics
Diferent instructors employed diferent practices regarding assigning marks for the use of clickers, as shown in Table 1.hese data are consistent with what others have suggested (e.g., Crouch & Mazur, 2001;Dufresne et al., 1996); most instructors in this study used clickers to encourage participation; far fewer (19%) used them as an assessment tool.
While most instructors prepared clickerquestions before lectures, 50% of interviewees reported occasionally thinking of a clicker-question in the middle of a class and asking it.As others have suggested (e.g., Crouch & Mazur, 2001;Dufresne et al., 1996), this had the efect of livening things up, enhancing ideas, and helped in clarifying topics, but instructors in this study reported that for the spontaneous questions to be efective, they had to be simple.

Table 1
Some Results of the Faculty Interviews (n=32)

Do you give students marks for the use of clickers?
No marks at all; clickers voluntary 31% Yes, for participation and / or clicker registration only 50% Yes, and correct answers contribute more to marks than incorrect answers 19%

What types of clicker questions do you ask?
Conceptual 84% Fact checking 78% Ones which do not necessarily have a right answer 56% Surveys about the class 50% Indication of student conidence in answer 13%

Do you expect / encourage / allow students to discuss a clicker question amongst themselves before / after they vote?
Yes, both before and after 63% No, neither before nor after 13% Yes before vote, but not after 15% Not before vote, but yes, after 9% he clicker polling procedures that instructors reported in this study resemble a variety of procedures that others have discussed in the literature (e.g., Beatty, 2004;Crouch & Mazur, 2001;Rao & DiCarlo, 2000).Instructors typically gave students between 30 seconds and one minute to answer a question before closing the voting.If calculations were involved, longer times, such as up to two or three minutes were allowed.All but one of the interviewees tended to show the class a histogram of the results of each vote immediately after the voting for a question was closed.About one third of the interviewees had occasionally showed the class a histogram of results during the vote, so that the students could actively change their answer and to see the efect on the histogram in real time.his introduced the potential of a histogram inluencing a vote.One economics professor reported using this technique to teach about 'herding' and to emphasize the value of independent thinking.
As shown in Table 1, the majority of interviewees said that they often expected, encouraged, or allowed students to discuss a clicker question both before and after voting.Of the faculty who did not allow discussion before the vote, this was usually because the correct answer counted for marks.One instructor regarded discussion as cheating, but felt that "it was probably happening anyway."

Pedagogical practices
Faculty were asked what types of questions they used with the clickers; the results are shown in Table 1.
Consistent with what others have discussed (e.g., Reay et al., 2005), the majority of interviewees responded that they used clickers to ask conceptual questions;12.5% of interviewees said they sometimes asked a question, and then, before giving the answer, asked the students to report their level of conidence in their own answer.
For questions that do have a correct answer, not all faculty were expecting or hoping that a large majority of students would get the correct answer; 34% of interviewees indicated that they were aiming for approximately 50% correct answers, citing peer instruction as a motivation for striving for this average.A lower correct response rate seemed to promote the vote − discuss − then vote again process described above.
We asked faculty who were new to clickers about what changes they might make to their pedagogy if they were to use clickers in the future.Most responded they would put more efort into formulating questions, include more conceptual questions as opposed to fact-checking, and encourage discussion before the vote.Some instructors were planning novel ideas, for example incorporating animations, graphs and math tools to teach various concepts such as game theory (e.g., prisoner's dilemma).Many of the faculty interviewed in this study reported that they had not previously thought about many of the issues raised in the interview, and said they would change their future teaching practices with clickers based on our interviews.

Advantages of clicker use
Consistent with what others have reported (Beatty, 2004;Burnstein & Lederman, 2003;Carnevale, 2005;Crouch & Mazur, 2001;Dufresne et al., 1996;Mazur, 1997;Rao & DiCarlo, 2000), the most common advantage of clicker use, reported by 69% of our interviewees, was that of student engagement.By using clickers, students are forced to think and make a decision in class, and this helps to engage them with the material.
Of the other advantages reported, the most common were: • The instructor receives quick feedback on student understanding of course material.• The students receive quick feedback on their own understanding and how they compare to the rest of the class.• Clicker use helps stimulate in-class discussion and peer instruction.
• Clickers engage all students equally, including the quieter ones who would not normally be involved in a spoken discussion.

Disadvantages of clicker use
Again, consistent with the literature (e.g., Beatty, 2004;Burnstein & Lederman, 2003;Fies & Marshall, 2005), the most common disadvantage of clicker use reported by faculty was the administrative burden associated with the technology.his included registering student identiication with clicker frequency, enforcing policies about lost or forgotten clickers, and tabulating and posting clicker marks.
Other common disadvantages reported were the extra time and energy instructors needed to devote to lecture preparation in order to use clickers efectively, and the fact that stopping the class for a clicker vote takes away from class time, so that less material can be covered.
Most instructors agreed that they would not use clickers in small classes, such as those with fewer than 30 students.he administrative burden and other disadvantages outweigh the advantages in these small classes.In larger classes, such as 70 or more students, the advantages are felt to outweigh the disadvantages.

Correlations between Teaching Practices and Student Experience
We asked all of our faculty interviewees if they would survey students in the classes in which they were using clickers.Students were asked whether they liked using clickers, and whether they believed using clickers helped their learning.hese surveys were conducted in class using clickers.In a pilot student survey, involving three classes with a total of 670 students, responses were simply phrased as yes/no.he majority of students in all three classes said "yes" to both questions.In a larger student survey, involving 6 classes with a total of 715 students, a 4-point scale was used to indicate the level which students liked the clickers and the level which they thought clickers helped them learn.he data from the 4-point scale were collapsed to match the binary (yes/no) scale of the pilot data. he results for all nine classes are shown in Table 2.
he results of the student surveys in all of the 9 classes were compared to some of the reported teaching practices of the interviewees.We performed logistic regression analyses to determine the likelihood of students reporting that "yes", they like using clickers and that "yes" they believe clickers help them to learn, given certain teaching practices, as reported in the faculty survey.he student data were compared to the data of the following four binary (yes/no) questions posed to the instructors of these nine classes: 1. Do you expect/encourage/allow students to discuss a clicker question amongst themselves before they vote?2. Do you ever think of a clicker question in the middle of a class and ask it?3. Do you ever display a histogram of vote results while voting is going on so the students can see the results while they can decide on or change their answer?4. Do you ever have students discuss a clicker question after they have voted?
he likelihood of being able to predict students liking clickers and believing that clickers help with learning based on instructors teaching practices was evident in three of the four teaching practices questions.As shown in Figure 1, it is likely that students will believe that clickers help their learning if instructors allow them to discuss a clicker question amongst themselves before they vote.As shown in Figure 2, it is likely that students will believe that clickers do not help their learning if instructors allow students to discuss a clicker question after they have voted.cussion of questions before a vote is helpful to their learning, but that discussion of questions after a vote is not helpful to their learning.A possible explanation for this might be that students perceive this post-vote discussion to be frivolous and this takes away from class time that could be used to cover more material.Also suggested here is that students believe they will not learn more just because instructors display the histogram during a vote.Again, it might be that students regard this practice as a frivolous use of the technology, and again, detracting from time and focus in the lecture that could be spent going over different material.

Conclusions
here are many ways to use clickers in the classroom, as well as many reasons to use or not use them.As demonstrated in this study, and as Beatty (2004), Brueckner and MacPherson (2004), and Roschelle, Penuel, and Abrahamson (2004) have also reported, most often, students like them.For students, the value of using clickers is greatly determined not only by how, logistically, the technology is used, but more importantly, how and why, pedagogically, it is used by the instructor.Beatty (2004), Brueckner andMacPherson (2004), andMazur (1997) also suggest this.

Table 2
Results of Student Surveys in Nine Classes