Sequential Writing Assignments to Critically Evaluate Primary Scientific Literature

In the later years of undergraduate study, students read, process, and evaluate primary literature within specific fields of study. Shifting from textbooks to the vast amounts of peer-reviewed current literature can be difficult for students. This article details an innovative approach to helping students successfully make this transition through a series of sequential assignments based on the levels of increasing cognitive complexity in Bloom’s taxonomy. These assignments are designed to be flexible enough to use in fields throughout the sciences and beyond, while allowing instructors to tailor these assignments to meet the needs of their particular students. Dans les dernières années de leurs études de premier cycle, les étudiants procèdent à la lecture, à l’assimilation et à l’examen des principaux travaux dans des domaines d’études particuliers. Le passage des manuels d’apprentissage à la masse profuse des travaux de recherche actuels évalués par les pairs peut être difficile pour les étudiants. Dans notre article, nous présentons une approche novatrice visant à aider les étudiants à réussir cette transition grâce à un ensemble de devoirs dont la succession répond à l’échelle de complexité cognitive selon la taxinomie de Bloom. Ces devoirs sont conçus de manière flexible pour un usage dans différents domaines scientifiques et ailleurs, ce qui permet aux enseignants de les adapter selon les besoins particuliers de leurs étudiants.

In the later years of undergraduate study, students read, process, and evaluate primary literature within specific fields of study. Shifting from textbooks to the vast amounts of peer-reviewed current literature can be difficult for students. This article details an innovative approach to helping students successfully make this transition through a series of sequential assignments based on the levels of increasing cognitive complexity in Bloom's taxonomy. These assignments are designed to be flexible enough to use in fields throughout the sciences and beyond, while allowing instructors to tailor these assignments to meet the needs of their particular students.
Dans les dernières années de leurs études de premier cycle, les étudiants procèdent à la lecture, à l'assimilation et à l'examen des principaux travaux dans des domaines d'études particuliers. Le passage des manuels d'apprentissage à la masse profuse des travaux de recherche actuels évalués par les pairs peut être difficile pour les étudiants. Dans notre article, nous présentons une approche novatrice visant à aider les étudiants à réussir cette transition grâce à un ensemble de devoirs dont la succession répond à l'échelle de complexité cognitive selon la taxinomie de Bloom. Ces devoirs sont conçus de manière flexible pour un usage dans différents domaines scientifiques et ailleurs, ce qui permet aux enseignants de les adapter selon les besoins particuliers de leurs étudiants. e hope to provide our undergraduate students with skills that will be useful to them in both post-graduate studies as well as the workforce, in addition to helping them develop a strong fact base in a specific discipline of study. In the early years of undergraduate education, we tend to focus on bolstering that fact base using packaged and contextualized information that is found in textbooks. In later years, we ask our students to shift to primary literature, analyzing information in a critical manner that models the type of reasoning we hope students maintain long after graduation. However, this transition can be challenging for students. Primary literature tends to be laden with jargon that is discipline specific, rather than incorporating a writing style aimed at a general undergraduate audience, as we find in textbooks. Students may also find the sheer volume of primary literature available to be daunting and may have difficulties selecting which resources to trust and incorporate into their own work.
As instructors of students facing these challenges, we should be asking ourselves how to work through this informational transition to enhance our learners' critical thinking skills that, while practiced in the classroom, will translate to critical evaluation of ideas outside of the classroom. For example: How are students to evaluate the disciplinary importance and validity of a study? Can students be taught to discern important disciplinary findings (i.e., findings that become integrated in future textbooks) and the findings that will rarely be W discussed again? Most importantly: Is there a way we can help guide each of our students to this level of understanding?

Previous Methods to Bolster Undergraduate Critical Reading
Building undergraduate critical reading skills is an ongoing challenge within the pedagogical community. As a result, several methods have been developed to guide students through reading original literature with varying degrees of scaffolding and emphasis on different aspects of the articles. Some particularly useful methods include Figure Facts (Round & Campbell, 2013), CREATE (Hoskins et al., 2007), structured reading questions (Van Camp & Van Camp, 2013), and New York Times style writing assignments (Brownell et al., 2013). Figure Facts is a method that directs students to fill out a detailed table, which consists of questions that focus on the figures of a scientific journal article (Round & Campbell, 2013). The emphasis of this technique is to help students move away from reading journal articles as they might read textbooks by emphasizing the text over the figures. This more closely mimics the style of discussion of original research in graduate school and beyond, in which "going through the figures" is proxy for discussing the entire paper. An advantage of this method is that students increased the time spent examining and taking notes about the figures, which resulted in students being more skilled in interpreting data and feeling less frustrated when reading primary research articles. A difficulty with this technique is that it may be difficult to generalize to fields outside of the specific sciences in which data figures are the key method for conveying data in original research (compared to social sciences, e.g.). This also leaves unaddressed the struggles students can face when reading other sections of papers, such as the introduction and discussion.
Another method involves designing a course to revolve around a series of journal article assignments (Hoskins, Stevens, & Nehm, 2007). The CREATE (consider, read, elucidate hypotheses, analyze and interpret the data and think of the next experiment) method follows the path of a single line of research throughout the term, allowing students to both practice interpreting original research while simultaneously developing a better appreciation for the way science progresses. There are clear benefits to this method, as students demonstrated improvement on pre-and post-course tests of critical thinking about data and reported generally positive feelings about the experience (Hoskins et al., 2007). Another version of CREATE was shown to reduce the number of students who failed to complete a general education neuroscience course (Bodnar et al., 2016). The major hindrance to generalizability is that this technique necessitates reading journal articles that all revolve around a single topic. This may be limiting for instructors whose courses range in topics more broadly.
General critical reading assignments have also been incorporated in the classroom, with questions focused on locating the thesis of a work and listing supporting evidence. One study examined performance on pre-and post-course tests of critical reading skills after a series of structured reading questions, and found improvement, with students reporting they found the reading questions helpful (Van Camp & Van Camp, 2013). As the content of the tests were not counterbalanced, and there was no control group for comparison, the findings of this study are difficult to interpret. However, the principle of repeated practice of critical reading throughout a term is sound, and this work can be viewed as another example for educators to draw from.
Another course used frequent writing assignments and in-class discussions to enhance scientific understanding (Brownell et al., 2013). The writing assignments were modeled after the style of New York Times summaries, with repeated rounds of drafts and feedback built in throughout the term. Preand post-course questionnaires revealed that this method enhanced students' confidence in comprehending and communicating about scientific research. The concern with this method is its scalability, which is dependent upon departmental resources and support from graders or teaching assistants. However, alternate versions of this assignment with fewer rounds of feedback may also provide beneficial results and could be used in larger classes with fewer resources.
In sum, there are a number of outstanding resources already developed to assist with primary literature comprehension. However, these resources are not always a good fit due to practical limitations such as the generalizability across disciplines and time constraints that affect the number of assignments that can be included within a course. The new method presented here is a series of assignments designed to be flexible in numerous contexts (e.g., student experience level, class size, discipline) and can be used progressively or in isolation.

Bloom's Taxonomy and Critical Reading
These assignments model the levels of increasing cognitive complexity in Bloom's taxonomy (Anderson & Krathwohl, 2001) starting from basic comprehension and continuing through to the creation of new ideas in order to create a method that is useful for a range of skill levels. Bloom's was used as a framework to design these assignments because it is a model we use frequently to envision how our learners progress from novice to expert levels of understanding. Thus, it is an empirical question as to whether or not the level of difficulty is truly increasing along the progression of assignments.
Categorizing the assignments by their relevance to the different levels of Bloom's taxonomy also allows an instructor to quickly gauge which assignments might be most appropriate (see Table 1). In this way, if an instructor's course were to consist primarily of novice psychology students (e.g., lowerdivision, or first-or second-year courses), the instructor may choose to use the "Understand" and "Apply" assignments, and forego the rest. Likewise, if an instructor were to teach more advanced students, they may choose to start with the "Analyze" assignment, followed by the "Evaluate", and/or "Create" assignments, while skipping the "Understand" and "Apply" components. A mixture is likewise possible, in which an instructor has their students provide a summary of an article ("Understand") followed by a critique ("Evaluate").
For example, in one of my third-year courses, I start out by modeling how to write a summary in class, emphasizing the distillation of important concepts and the challenge of writing succinctly. Students are then left to write a summary and an article comparison ("Analyze") for one of several assigned journal articles before the midterm. The assignment is due before the class in which the article is discussed, so that students may better showcase their own level of understanding and analysis in their written assignments. Having multiple articles to choose from as well as multiple due dates allows students to pick an article to analyze that they find interesting, or, alternately, pick an article to analyze whose due date is during a week that is lighter in terms of coursework in their other classes. I highly recommend this kind of small intervention that allows more flexibility to our students. They appreciate the choice, and I appreciate the ability to grade a large number of assignments over the course of several weeks rather than all at once. My students receive feedback on their work and use it to guide their summaries and critiques due during the second half of the term.

Description of Assignments Understand
One of the first challenges for students learning how to read and distil primary literature is understanding the major implications of the work. Students may initially focus too much on details such as the individual references cited that, while relevant, are not central to the point of the paper. This assignment is a first step at challenging students to understand the importance of a paper. They are required to cut through jargon and succinctly summarize the main points of an entire paper. The skills students develop with this first assignment provide a solid foundation upon which to develop more sophisticated analytic abilities.

Apply
The next challenge for our students to overcome is applying information they have learned to a novel situation. The key to this skill is to think about the deep structure of a problem that was addressed in a paper and to use the solution in a new situation. The level of similarity between the "old" and "new" problem can vary to match your students' needs. Many instructors are already incorporating this idea into their tests, but using this as a writing assignment could allow a more detailed, thoughtful response from students.

Analyze
Once students are able to understand and apply knowledge presented in primary literature, our next challenge is to help students recognize that they have enough knowledge and skill to form an opinion about the quality of original research. Before jumping into a critique of a single paper, a structured comparison of two papers can help train students to identify qualities of papers that help determine how "strong" or "good" they are. This concept of the strength of a paper can be challenging to convey to students but is worth spending time exploring. A strong paper addresses an important topic, has excellent argumentation and methods throughout, and is ultimately more likely to be referenced and discussed in the future. Instead of first asking students to determine whether or not a paper is strong, we can ask if one paper is stronger than another. For this skill to be developed, the two papers do not need to be directly related. Assessing which paper better motivates its study in the introduction, or which paper better matches its techniques to its questions of interest are decisions students can make about two papers on disparate topics. While the comparison (Analyze) builds up the skills necessary to identify weaker versus stronger pieces of research, the critique allows the students to take a broader view and judge whether a single article is weak or strong, overall. I sometimes tell my students to think of the critique as a comparison between one paper and all other papers out there, or the Platonic ideal of a paper. This is the skill we hope students practice when distilling the large amounts of previously published work in their literature reviews or research proposals. This skill will also serve them well after graduation when determining the value of novel research findings reported in popular media and elsewhere. I highly recommend building in some steps before the critique, as many students initially feel as though they are not qualified to critique a published paper.

Create
Finally, after students are able to successfully critique previously published studies, they are ready to create their own. This skill can be practiced either through a more traditional research proposal or by giving students a section of a paper (e.g., the methods) and asking them to write up the remaining sections. Excelling in this skill involves a detailed understanding of the previous work in a field, as well as an understanding of what makes a "good" paper. Without a solid foundation, students are apt to submit proposals for replications or very mild modifications on experiments already discussed in class. Creativity is something even seasoned scientists work on cultivating throughout their careers. Using the preceding assignments to develop fundamental skills in understanding and critiquing research will assist our students to develop their own, novel research ideas.

Conclusions
Most educators are hoping to assist their students in developing critical thinking skills that can be applied both in and out of the classroom. In the short term, these skills can help students succeed in their undergraduate programs. In the long term, these skills can help students become better informed citizens. Developing these skills is a multifaceted process that should take place in stages throughout the years of postsecondary education. Using Bloom's Taxonomy, the assignments described here model one way of incorporating these skills throughout undergraduate coursework. If one or more pieces of these assignments sound like they could be useful additions to your classroom, please refer to the Society for the Teaching of Psychology website (http://teachpsych.org) in 2020 for the full set of assignment descriptions and rubrics.