A Post-Class Question Strategy That Provides Feedback and Connects In-and Out-of-Class Learning

M any strategies have been developed to actively engage students in large classes (Boehmler & Smith, 2006; Caldwell, 2007; Cameron, 1999; Clouston & Kleinman, 1999; Cutts, Kennedy, Mitchell, & Draper, 2004; Harley, Maher, Henke, & Lawrence, 2003; Harpp, 1994; Lyon & Lagowski, 2008; Rutherford, 2004); however, it remains challenging to provide students with relevant, individualized feedback, to emphasize the value of time on task (Chickering & Gamson, 1987), and to connect inand out-of-class learning. Recently, online organic chemistry homework programs (ACE Organic: Achieving Chemical Excellence; Connect: Teaching and Learning Program; OWL: Online Web Learning; Reaction Explorer: Organic Chemistry Tutorials; WileyPlus: Courses in Organic Chemistry) have been developed that provide some important advantages over other homework software. The programs have questions that ask students to draw their own answers (molecules), which provides a valuable addition to multiple choice, matching, ranking, and numeric questions. The software is capable of molecular recognition (i.e., of recognizing the molecule that the student has drawn and comparing it to the correct answer), which allows the program to give tailored feedback that is based on the student’s own answer. Using the gradebook, the instructor can review the answers provided by each student, thereby obtaining information with respect to the students’ level of understanding. 27

Recently, online organic chemistry homework programs (ACE Organic: Achieving Chemical Excellence; Connect: Teaching and Learning Program; OWL: Online Web Learning; Reaction Explorer: Organic Chemistry Tutorials; WileyPlus: Courses in Organic Chemistry) have been developed that provide some important advantages over other homework software.The programs have questions that ask students to draw their own answers (molecules), which provides a valuable addition to multiple choice, matching, ranking, and numeric questions.The software is capable of molecular recognition (i.e., of recognizing the molecule that the student has drawn and comparing it to the correct answer), which allows the program to give tailored feedback that is based on the student's own answer.Using the gradebook, the instructor can review the answers provided by each student, thereby obtaining information with respect to the students' level of understanding.An instructional method is described that was used in a large, introductory organic chemistry course to ask online, post-class questions and to use the students' answers to design the lesson and learning activities for the following class.Additional goals of this method were to provide relevant, regular, and prompt feedback to students and the instructor, to emphasize the value of time on task (i.e., that learning takes time) and to connect in-and out-of-class learning.Herein, the results of formative project evaluation and preliminary results of an educational evaluation of the method are also described.
Even though I assigned weekly quizzes from one of the aforementioned homework programs (ACE Organic: Achieving Chemical Excellence) and reviewed the more difficult questions in class, I felt that the quizzes were somewhat disconnected from the in-class work.This disconnection was especially evident when the difficult questions were reviewed long (up to two weeks) after the topic had been covered in class.Furthermore, and not surprisingly, discussion forum statistics (where students asked many quiz-related questions) peaked in the hours before the quiz was due.This surge in participation suggested that students were not reviewing the concepts discussed in-class immediately following the class, which would be ideal to cement their learning (Chickering & Gamson, 1987).
To address these concerns, I developed a post-class question method with the intent of satisfying the following objectives: 1) to encourage students to spend time on task; 2) to connect in-and out-of-class learning; 3) to provide prompt, regular, and relevant feedback to the students; and 4) to provide feedback to the instructor.This method is based on scaffolding learning theory, which involves supporting students through the learning process by appropriately sequencing content, tasks, and teacher and peer support until students can apply new skills and strategies independently (Larkin, 2002).Herein, I have described the method and the initial results of the evaluation of the method's effectiveness.

Background
The courses described herein, Organic Chemistry I and Organic Chemistry II, are large (250-420 students per section), first-and second-year organic chemistry courses at the University of Ottawa.They are generally considered to be challenging courses.Approximately 67% of students enrolled are from the Faculty of Science, 21% from the Faculty of Health Sciences, 6% from the Faculty of Engineering, 5% from the Faculty of Social Sciences, and less than 1% from other faculties such as the Faculty of Arts.Organic Chemistry I and II are required courses for the majority of students' enrolled and are required for admission to many graduate programs and professional schools, such as medicine.
Each course involved weekly quizzes and twice weekly post-class questions using ACE Organic, an online homework program (ACE Organic: Achieving Chemical Excellence), clickers (Bhattacharyya, 2006) (except in 2008 when clickers were not used), two midterms and a final exam.The first-year course also had a three-hour laboratory component.In each course there were two 80-minute lectures per week, optional 80-minute tutorial sessions, and optional written assignments for which answers were posted on the course website.

Description of the Method
I assigned an online post-class question after each class; an example is shown in Figure 1.The question probed the students' understanding of a subject and/ or concept taught in that class.The students had two attempts to answer the question and the program provided feedback based on their response.In the first two courses (while the method was being developed) the post-class questions were worth bonus marks only (up to 1% of the student's final mark).In 2010 and 2011, post-class questions were worth 2% of the final grade.The student was accorded 100% for a correct answer, 90% for an incorrect answer, and 0% if he/ she did not attempt the question.
I reviewed the students' answers prior to the following class, and used these answers to design the first segment of the next class.If most of the students had obtained the correct answer, then we spent very little class time, if any, discussing the topic related to the post-class questions.However, if students struggled to answer the question, then I used the students' answers to design a clicker question, typically in multiple-choice format, which contained the correct answer along with the most common incorrect ones.If students struggled to answer the clicker question, then I asked them to pair up and to discuss the problem.I asked the clicker question a second time and the results of that question drove an in-class discussion, which concluded that section.An overview of the method is shown in Figure 2.

Figure 2
Online post-class question method I typically spent 5-10 minutes creating each question, 2-10 minutes reviewing students' answers, and 5-10 minutes to creating a new clicker question (which I did for approximately two thirds of the classes).At the end of the semester, I spent about 20 minutes calculating the grades by exporting the data from the ACE gradebook to an excel spreadsheet then assigning the marks as described above.
I started assigning post-class questions a few weeks into each class (to give students time to get used to the course and its requirements) and, subsequently, at least one post-class question was assigned after every class.Students could answer a question correctly in as little as two minutes, depending on their levels of proficiency and commitment to reviewing/learning the lesson in question.Unfortunately, ACE organic does not provide statistics on the amount of time taken for a student to complete a question/assignment.Students could also review and/or re-do the questions as part of the preparation for midterms and final exams.

Project
I designed an evaluation of this project, which is currently underway, largely based on the format suggested in the National Science Foundation's 2010 User-Friendly Handbook for Project Evaluation (Frechtling et al., 2010).The first phase of the evaluation involved a formative evaluation of the method; the second phase is a summative evaluation.The results of the formative evaluation phase are described below, which sought to answer the following questions: 1) What was the participation rate in each class?2) What were the students' opinions and suggestions with respect to the aforementioned project objectives? 3) What were the instructor's opinions and suggestions with respect to the aforementioned project objectives?The summative evaluation phase is currently underway, in which the key research questions are: 1) What is the degree of improvement in learning outcomes, if any? 2) What is the most effective way of using the post-class question method?The preliminary results of the summative evaluation are described below.

Participation rates
The participation rates varied from 61.4% to 75.1% (Table 1) and were higher in the second-year course than in the first-year course.In 2010, I increased the weighting of the post-class questions to 2% of the students' final grade; they had previously been worth bonus marks only.There was a concurrent improvement in participation rates on post-class questions when comparing rates between different cohorts of the same course (Table 1).In every course, at least 25% of the class did not participate.While the reasons for this have not yet been explicitly probed, it is perhaps because of the low weighting of the post-class questions (bonus marks or 2% of the students' final grade).Class attendance did not change significantly with the addition of post-class questions, which could be attributed to the fact that clickers were already in use in the class (for which participation marks were assigned), the fact that lecture notes were not available online for the course (perhaps providing motivation for students to come to class), and/or other reasons.
The feedback received on student surveys has been extremely positive.A selection of students' responses is shown in Figure 3, below.Specifically, students responded that the post-class questions helped them learn class material, that they reviewed their notes regularly in order to answer the post-class questions, and that they gave a reasonable effort to respond to the questions.
The students' comments were also very positive.For example students said that: "Post-class questions really forced me to review the stuff i [sic] learned in class, so i think it a great idea.""Also going over the [questions] which most people had trouble with in class was also good, because then we could see exactly where we went wrong.""The post-class questions were great to encourage at least a little 'daily' reviewing, and again, I think it's great that we were rewarded just for trying." Very few technical difficulties were reported, the main one being that the program was "picky" and so a slight error in the student's answer, which might be unrelated to the student's understanding of the concept being tested, was nevertheless recorded as incorrect.Most students learned to draw organic molecules on a computer for the first time in Organic Chemistry I and they could do so through online tutorials before attempting an assignment.Fewer than five students per course reported technical difficulties related to drawing/submitting answers.

Figure 3
Selected student survey results from Organic Chemistry I, 2009 (N=604)

Instructor's opinions and suggestions
Overall, I feel that the time required to create the questions, review students' answers, and design follow-up questions/discussion topics was well worth the effort.It was helpful to see students' answers regularly instead of just after midterms/ exams and to be able to address common errors so easily and often.It also seemed that student engagement had improved in the class, which was particularly observed with more students asking questions about the last class' lesson and having more students contributing to in-class discussions (anecdotal, qualitative results).
The largest technical difficulty on my part arose with the release time of the questions (i.e., when the question became available to students).With the program that we were using, I could not specify a specific time to release the question, and so there was occasionally a delay between the end of the class and the posting of the question, depending on how long it took me to get back to a computer.This was a very minor problem and for the most part the post-class questions were straightforward to assign and review.

Comparison of learning outcomes
Preliminary results at the question level suggested that learning outcomes have improved.For example, Table 2 shows a post-class question (for which answer choices were not provided) in which I asked students to draw the first organic intermediate for the reaction shown.Initially, only 27% of students obtained the correct answer.When I asked the same question at the beginning of the following class as a clicker question, with the correct answer along with the most common incorrect answers shown, 55% of students obtained the correct answer (t(766) = 8.441, p < 0.0001).After peer discussion, 88% of students gave the correct answer to the same question (t(652) = 10.826,p < 0.0001).Although the same question was asked each time, Mazur has shown that students often arrive at the correct answer after peer discussion, especially when 30 -70% of students have initially answered correctly, and even if the correct answer was not originally the predominant answer (Mazur, 1997).
A comparison of the final exam results between 2008 and 2009 for question types (S N 1) that were similar to the one shown above (Table 2) also suggested an improvement in learning outcomes.While the question shown in Table 2 was given as a post-class question in the Organic Chemistry I class in 2009, a question of this type was not asked as a post-class question in the Organic Chemistry II class in 2008.An approximately equal amount of time was spent in class on the topic on this topic (i.e., the S N 1 reaction) in both courses.In 2009, 80% of students were able to draw the correct answer on a different question of the same type as the post-class question.This was in contrast to the 67% of students in 2008 who were able to draw the correct answer (t(1281) = 5.356, p < 0.0001).
A correlation between post-class questions and grades, but not necessarily that the post-class questions have caused the improvement in grades.There was no statistically significant difference in the students' final grades or GPAs between cohorts that used post-class questions and those that did not (which was provided to the author, stripped of all student identifying information, by the Institutional Research and Planning office at the University of Ottawa).This was not surprising given the number of factors that can affect a final grade or GPA, particularly between different courses or different cohorts of students.Nevertheless, the promising results thus far are encouraging and more investigation is merited.

Concluding Remarks
Asking online post-class questions after every class encouraged students to review each class immediately upon its completion and the online homework program provided them with feedback tailored to their individual answers.I used this method to gauge the degree of students' understanding after every class, identify common errors, and tailor subsequent classes accordingly.Preliminary project evaluation results suggest that this method had a positive impact on student learning, although there was not a statistically significant difference in students' final grades following the introduction of the method.The merits of this method are currently being more fully evaluated.
Figure 1A post-class question with feedback for an incorrect response.Source: ACE Organic.Used with permission.

Method
Distribution of results in response to the question: "Draw the first organic intermediate formed in the reaction shown below." Organic Chemistry I, 2009 exam question (N=606) a,c 80 Final exam question, 2008 (N=680) c,d 67 a Data from 2 sections.b Data from 1 section.c The question was a variation on the same topic.d Organic Chemistry II (Data from 2 sections).