What are Students ' Learning and Experiences in an Online Learning Tool Designed for Cognitive and Metacognitive Skill Development ?

Organic chemistry has its own representational language and as with any language, any barrier with the language makes understanding the message difficult. In this article, we describe the design of an open education resource that empowers students to master this language. We also describe the structure and findings of a pilot study designed to measure students' learning and experiences using the module, with a focus on principles, structure, and key findings, rather than on the chemistry-specific elements.

Organic chemistry has its own representational language and as with any language, any barrier with the language makes understanding the message difficult.In this article, we describe the design of an open education resource that empowers students to master this language.We also describe the structure and findings of a pilot study designed to measure students' learning and experiences using the module, with a focus on principles, structure, and key findings, rather than on the chemistry-specific elements.
ave you ever tried to learn a new language?What tools did you use to learn that language?How long did it take you to learn?What challenges did you face?Perhaps you took a course or used an app to help you practice.Maybe you practiced conversing with others in attempts to become more fluent.If it was a language you were somewhat familiar with, you may have been able to learn it more quickly than a language largely different from your own.Organic chemistry has its own language and as with any language, barriers exist that make it difficult to understand a given message.Organic chemistry's language comprises symbols and representations, which communicate the macroscopic level and submicroscopic levels of organic chemistry (Johnstone, 1982(Johnstone, , 2000;;Taber, 2013;Talanquer, 2011) (Figure 1).The submicroscopic level, which we cannot see through an ordinary light microscope, can be seen through changes of the macroscopic level in chemistry all around us every day (Johnstone, 1982).For example, when we boil water we do not see the individual molecules; however, we can see the visible changes from still water to bubbling water as molecules move from a liquid to gaseous state.Students struggle, however, with organic chemistry's symbolism (Kozma & Russell, 1997) and in order for students to learn organic chemistry concepts, they must learn how to speak and understand the language (Taber, 2009).Students may struggle to become H DOI: 10.22329/celt.v11i0.5039fluent in organic chemistry's language because of its technical and specialized vocabulary in addition to the symbolism (Bhattacharyya, 2013;Galloway, Stoyanovich, & Flynn, 2017).In this article, we describe an open education resource that empowers students to master this language, the Organic Mechanisms module in orgchem101.com.We also describe the methods and findings of a pilot study designed to measure students' learning and experiences using the module.

About OrgChem101.com
OrgChem101.com (OrgChem101) is an online, bilingual (English and French) tool created to help students learn organic chemistry's language and core concepts and provide students with the tools to help them better interpret organic reaction mechanisms, as well as improve their metacognitive skill (Azevedo, Behnagh, Duffy, Harley, & Trevors, 2012).OrgChem101 is free for anyone to use and includes three modules: Organic Nomenclature, Organic Mechanisms, and Acid-Base Reactions.
OrgChem101 is designed to facilitate learning organic chemistry concepts through guided, interactive activities that allow students to learn the theory and put that knowledge into practice at their own pace, allowing the learner to be in control (Figure 2).To help students improve their metacognitive skills, two of the modules feature immediate feedback and built-in metacognitive exercises.Cost and access can be barriers to students, which is why OrgChem101 is an open education resource (OER).OrgChem101 allows learners to access content anywhere with an Internet connection and allows them to practice and gain feedback from the comfort of their own home.Lastly, OrgChem101 is scientifically accurate thanks to reviews from experienced professors and organic chemists; learners can be assured that the content they are learning and the feedback they are getting is correct.

Figure 2
Features of OrgChem101.

Organic Nomenclature Module
The organic nomenclature module, called Nomenclature101, was created first and the impact of using this module has been studied the most extensively (Bodé, Caron, & Flynn, 2016;Flynn et al., 2014).Students learn organic nomenclature to communicate and interpret more complex concepts, including functional group names (Flynn et al., 2014).Students who participated in a nomenclature workshop demonstrated significant learning gains with both guided and independent use of the nomenclature module (Bodé et al., 2016).Additionally, students found the nomenclature module useful and easy to use and reported that the module promoted an enjoyable learning experience (Bodé et al., 2016).

Organic Reaction Mechanisms
The next two modules were designed building off our learning from the nomenclature module's design.To start the organic reaction mechanisms module (i.e., "Get started", Figure 3), students are prompted to complete a metacognition exercise during which they rate their ability to successfully answer questions aligned with each learning outcome (Figure 4), and then complete a pre-test to compare their actual score against their predicted score (Figure 5).The questions are aligned between the metacognition exercise and the pre-test.Once the metacognition exercise and pre-test are complete, students are prompted to assess their strengths and weaknesses and pinpoint areas for improvement.

Figure 3
OrgChem101.com organic reaction mechanisms module home page.

Information about the module
Go directly to a specific part of the module.

Figure 4
Organic Reaction Mechanisms module metacognition exercise.

Figure 5
Organic Reaction Mechanisms module pre-test.

Metacognition tacognition
Pre-test For each learning outcome, students are introduced to the skills through a video, for which the transcript is available (Figure 6).The videos provide interactive questions so that students can check their understanding as they go, before moving on to the practice questions.

Figure 6
Organic Reaction Mechanisms module learning outcome video.
Each learning outcome section contains many interactive, online practice questions specific to that learning outcome (Figure 7).Students can request help, go back or jump ahead throughout the module, submit their answers, and see the correct answers.

Figure 7
Organic Reaction Mechanisms module practice question.

Video Video Activities
What are the effects of using OrgChem101?
OrgChem101.com became publically available in Fall 2016, but the effectiveness and usefulness had not yet been determined.A pilot study conducted as part of an honours thesis project evaluated the effectiveness of and students' experience using the OrgChem101 organic reaction mechanisms module in helping students improve their learning experience and knowledge of organic reaction mechanisms, as well as students' metacognitive skill.More specifically, the study investigated the following research questions: 1) What are the learning gains, if any, when students use the OrgChem101 Organic Mechanisms module?2) What are students' experiences using OrgChem101 Organic Mechanisms module regarding its usefulness and ease of use? 3) How might the OrgChem101 Organic Mechanisms module help improve students' metacognitive skill?

Methods
The students enrolled in Organic Chemistry I were invited to attend a mechanisms workshop; Organic Chemistry I is students' first course in organic chemistry that lasts one-semester.The study received ethics approval from the University's Office of Research Ethics and Integrity.The workshop time did not conflict with the Organic Chemistry I class time and workshop participation was not mandatory, nor for credit.Before beginning the workshop, participants were asked to read through and voluntarily sign the form if they consented to being included in the study.Students were welcome to participate in the workshop without consenting to have their data used for research purposes.In total, 34 participants attended the workshop and all 34 consented to being included in the study.Participants were not informed of the website name upon recruitment; rather, the students were simply told they would work through an organic mechanisms module.Students might have accessed the module outside of the workshop, but previous experience is not expected to affect the study's findings During the workshop, participants completed a worksheet ("Worksheet 1") and then explored the mechanisms module for 30 minutes, during which participants could engage in discussion with other participants, followed by a second worksheet ("Worksheet 2") and a questionnaire (Figure 8).

Figure 8
Overview of the winter organic mechanisms pilot study.
The worksheets consisted of seven chemistry content questions and one metacognitive rating question (post-diction).The chemistry content questions focused on key aspects of organic chemistry's symbolism, specifically the static representation of dynamic chemical reactions.The metacognitive post-diction rating asked students how many questions they thought they answered correctly.All 34 participants completed the chemistry content questions on both worksheets; however, only 23 participants answered the metacognitive rating question on both worksheets and only 26 participants answered the questionnaire in full.The questionnaire consisted of 33 Likert scale questions with the options strongly disagree, disagree, neither agree or disagree, agree, and strongly agree.The questionnaire statements also combined both positively and negatively worded statements.There were also eleven non-Likert scale questions for students to provide their feedback.A paired t-test was performed in SPSS Statistics to analyze differences between Worksheet 1 and 2 scores.Likert scale questions were analyzed visually in excel and the feedback was analyzed qualitatively.The workshop was observed by two observers following the Classroom Observation Protocol for Undergraduate STEM (COPUS) to characterize how students and instructors spent their time during the workshop (Smith et al. 2013), and observations were analysed for inter-rater reliability.

Results and Discussion
Research Question 1: Worksheet Scores Decreased The mean scores for Worksheets 1 and 2 were 41% and 27% respectively, with a mean difference (Worksheet 2 scores -Worksheet 1 scores) of -14%, representing a statistically significant decrease, t(df) = 33, p <.0001 (Figure 9).One possible explanation for the decrease in worksheet scores is that students may have been fatigued by the end of the workshop.Without incentives, students may not have been motivated to put in as much effort into Worksheet 2.
Technical difficulties and database errors that students experienced during the workshop may have contributed to the lack of significant learning gains in the workshop as well.Students reported technical difficulties included problems with functionalities of the website (e.g., the submit answer button was not working on the day of the workshop), as well as database errors where the module would not recognize a correct answer or would provide the wrong answer.This could have led to participants' confusion regarding organic reaction mechanism concepts.We have since corrected the errors.
The short timeframe provided for students to work through the module may have also contributed to the lack of significant learning gains because they may not have had the opportunity to practice all the material that was tested on the worksheets.Future work should focus on students' continued use of the organic reaction mechanisms module.Additionally, the questions in worksheet two may have been considered more difficult than those in worksheet one, which may have contributed to the decrease in worksheet scores.
Although students had the option to engage in discussion while working through the module, COPUS data showed that students spent more of their time working individually than in groups.Future workshops may consider encouraging students to work together through the module.11).Furthermore, questionnaire responses indicated that students found OrgChem101 organic reaction mechanisms module easy to use and navigate (Figure 11).The fact that many students did not use or did not notice the help section (Figure 11) might also indicate that the module was easy to use and navigate.Students did not indicate that they tried unsuccessfully to find a help section.
Participants reported feeling comfortable using the organic mechanisms module and said they would use it again (Figure 11).More specifically, students found the video tutorials especially helpful in learning to draw electronpushing arrows and products.

Students enjoyed
OrgChem101's consistent interface and responded that the organic mechanisms module would lead to an improvement in their course grade (Figure 11).Furthermore, the majority of participants (26 out of 31) responded, in the non-Likert questions section of the questionnaire, that the organic reaction mechanisms module would have a positive impact on their success in Organic Chemistry I.

Figure 11
OrgChem101 organic reaction mechanisms module was useful and easy to use, as well as helped students perceive an improvement in metacognitive skill.
Too much time was allocated to use the module during the… The pace of the workshop was too fast I had a hard time navigating the module I prefer to study the material by myself The questions were helpful to draw the reverse reaction… The questions were helpful to draw the transition state The video tutorials were helpful to draw the reverse reaction… The video tutorials were helpful to draw the transition state The metacognition exercise was useful The module was a review from class The questions were helpful to learn to draw the products The module helped me improve my ability to assess my… The module will help improve my course grade The module's interface design was consistent The video tutorials were helpful to learn to draw the products The module's interface design made it easy for me to follow… Technical difficulties using the module obstructed my learning… The questions were helpful to learn to draw the electron-… The video tutorials were helpful to learn to draw the electron-… The module's website is well designed I felt comfortable using the module I will do this module again I like that I can use the module outside of class I appreciate that I can access the module anywhere that there…

Strongly Disagree
Neither Agree Disagree Percentage of participants' ratings (N=26) Research Question 3: Students More Accurately Predicted their Worksheet 2 Scores than Worksheet 1 (Metacognitive Skill) Of the participants who completed both Worksheet 1 and Worksheet 2 metacognitive post-diction ratings, 30.4% improved their metacognitive skill as indicated by their greater accuracy predicting their worksheet scores after the workshop than before (Figure 12).The study showed that although there were no significant learning gains after using the organic reaction mechanisms for a short amount of time in one workshop, there was an indication of an improvement in students' metacognitive ability.
According to questionnaire results, students explained that they determined their metacognitive post-diction rating based on how confident they felt, how well they recognized the material, and/or the amount of time it took them to complete a question.

Figure 12
Metacognitive rating was compared to the number of overall correct questions per participant.N = 23.The bubble size corresponds to the proportion of students who reported a certain metacognitive rating in relation to their number of correct questions.Data points above the y = x line represent students who rated their scores as better than they actually did (Figure 12).Worksheet 1 experienced a greater distribution of scores; however, more students fell on the y = x line for Worksheet 2, which shows students who correctly rated their performance to match their actual score (Figure 12).Students who fell below the y = x line underestimated their performance on the worksheets.

Implications for Instruction
As with learning any language, you can learn all the vocabulary you want, but becoming fluent in the feedback.As one student mentioned, "I feel like you can't learn this [organic reaction mechanisms] by just going to the lectures."OrgChem101 provides students with the practice of developing and conversing in organic chemistry's language, to then better understand the submicroscopic and macroscopic levels of organic chemistry.To help improve students' adoption of OrgChem101, the module could be implemented either in the classroom or in discussion groups directly.Professors could show students how to use the model effectively and emphasize the importance of developing metacognitive skill.OrgChem101 could also be integrated into courses by implementing it into the course assessments or used during class time with the professor's guidance.language is extremely difficult without practice and students' ability to learn organic chemistry concepts and metacognitive skillfulness is also important, as the results from this pilot study demonstrated a significant decrease in workshop scores due to a number of potential confounding factors, such as technical difficulties, fatigue, and differences in the level of difficulty between Worksheets 1 and 2. Currently, studies are being conducted using eyetracking experiments to further understand the functionality of the learning tool to allow for continuous improvement.The results from this pilot study have highlighted both some of the strengths and limitations in studying OrgChem101's effectiveness, which will be considered when expanded into a full study in the future.Evaluating OrgChem101's effectiveness and investigating students' experience with the learning module will be beneficial to continually improve the resources available to help students learn and understand organic chemistry concepts.

Conclusion
Overall, students found the Organic Chemistry Mechanisms module useful and helpful in learning organic chemistry content and metacognitive skill.Students have a desire to use OrgChem101 because it provides them with additional and flexible practice outside of the classroom.Professors may integrate OrgChem101 into their courses by encouraging students to practice at home using the learning modules or in tutorial classes, which would free up more time during lectures for class discussion and in-depth exercises.Continuing to study the effects of OrgChem101.com organic reaction mechanisms on