Exploring the Foundations for Student Success: A SoTL Journey

This paper illustrates the chronology of a research project which began in 2010 and continues today.  The research has evolved over time from a focus on the phenomenon (developing an understanding of student diversity and its impacts on student success), to experimental research (to learn the impact or benefits derived from the introduction of high impact practices), to a more complex understanding of the foundations for student success.  The fourth stage of the research, which is just underway, divides our efforts into two distinct directions.  The first is quantitative research utilizing institutional and learning management system data which was previously untracked and untapped.  The second is a shift to employing more qualitative research tools aimed at advocacy and institutional change.  Through each phase of the research the paper presents two distinct perspectives: First is the perspective of instructors-turned-SoTL-researchers as we muddle our way through understanding our challenges and learning how to use SoTL research methods to help guide the way.  The second perspective is that of an established SoTL researcher, who provides commentary and guidance to our journey.  Our hope is that the reader finds these two perspectives of a research journey both informative and valuable in providing insights into how a long-term research project might unfold.

This paper presents perspectives and outcomes from a multi-stage research project focused on improving understandings of foundations for student success.The research has shifted from a focus on the phenomenon (developing an understanding of student diversity and its impacts on student success), to experimental research (studying the impact or benefits derived from the introduction of high-impact practices), to a more complex understanding of the foundations for student success.In reporting on each stage, we combine two distinct perspectives: instructors-turned-SoTL researchers and established SoTL researchers.Our hope is that the reader finds this integration both informative and valuable in providing insights to how a long-term research project might unfold.
consensus has formed within growing circles in academia that there is scholarly research to be done on teaching and learning, that the systematic creation of rigorous knowledge about teaching and learning is a crucial prerequisite to responding to major challenges facing academia, that this knowledge must be shared publicly and should build cumulatively over time, and that the explorations of this area should be conducted by academics from all disciplines, not just those with appointments in schools of education.(Pace, 2004(Pace, , p. 1174) ) While written over a decade ago, Pace's vision that disciplinary academics engage in scholarly research on teaching and learning in their discipline remains a new and unexplored possibility for many people in higher education -for some it opens a gateway leading to exciting opportunities to explore questions about their teaching practices or their students' learning journeys, and for others it is a gateway leading to a strange and unfamiliar scholarly landscape that they A struggle to understand or value.Over the past decade, and longer, there have been several explorations of what motivates scholars of teaching and learning to enter the field of SoTL, or what has been called the teaching commons (Pace, 2004;Ashwin & Trigwell, 2004;Huber & Hutchings, 2005;Wuetherick & Yu, 2016).This paper tells the story of our journey -of both our research project and our teaching team's pedagogical strategies over the past eight years.We presented this information in a SoTL workshop at STLHE 2017, but we thought it would be best presented here as a story of our collective journey.The paper presents brief sections representing the stages of the project: a narrative of what we were experiencing and trying to do, and our collective insights into SoTL research and how lessons learned might enhance or streamline our continuing SoTL journey.
Through each phase of the research the paper integrates two perspectives: First is the perspective of instructors-turned-SoTL-researchers as we muddle our way through understanding our challenges and learning how to use SoTL research methods to help guide the way.The second perspective is that of an established SoTL researcher, who provides commentary and guidance to our journey.Our hope is that the reader finds this integrated viewpoint both informative and valuable in providing insights into how a long-term research project might unfold.

The University of Regina Context
Some context may be required to help the reader understand how our journey may differ from that of SoTL researchers at other Canadian institutions.At the University of Regina most of our students are either from nearby (southern Saskatchewan) or far away (International students from India, China, and many other countries).Comparing our student retention rates to others reveals a part of our story: in the MacLean's (2018) report on student graduation rates (students starting university in 2007 and graduating with a degree within seven years), University of Regina ranked near the bottom with 58.1% graduating, despite a first-to-second-year retention rate of 85%.These rates suggest an anomaly: the university welcomes students with a diverse mix of academic preparedness and finds pathways for retention beyond first year, yet over 40% of our students do not complete a degree.The goal of our research therefore was to develop an understanding of our students so that we could better support them in their learning journeys.

An Introductory Narrative: Frustration as a Catalyst for SoTL Research
This SoTL journey began in 2010 when a new introductory business course came on stream.Previously our students spent their first year in general studies.Many students struggled with core business classes in their second year of study.A 38% failure rate in an introductory accounting course was both frustrating and a motivator to learn more about student learning foundations.As instructors and novice SoTL researchers, we saw both a need for broader understanding of our students and an opportunity to shape pedagogical practice.We felt it a moral responsibility to better understand our students, and to develop the teaching tools which could improve success rates for all of them.
There are a number of motivating factors for faculty to engage in the scholarship of teaching and learning, but often it is rooted in a desire to answer a question about what is happening in their own classroom.Ashwin and Trigwell (2004) distinguish three levels on which SoTL occurs.According to their model, the purpose of the first level is to inform oneself about what is occurring in the classroom, and the methods and conclusions are verified primarily by the SoTL practitioner with the goal of 'personal knowledge' to improve their own teaching practices.It is at this level that this project was initiated.The purpose of the second level is to inform others within a shared context (within a Faculty, or discipline more broadly).At this level, the methods and conclusions are verified by others within that shared context, and it becomes what Ashwin and Trigwell (2004) call 'local knowledge'.And the purpose of the third level of SoTL is to inform a wider audience broadly across higher education, with the methods and conclusions becoming 'public knowledge' and verified by others across that wider audience.
This project began as a drive to enhance personal knowledge for one individual teacherscholar, but evolved to the second level of local knowledge as it began to involve others in the Faculty of Business teaching the core Business 100 gateway course.As the robustness of the research increased, and began to connect firmly with the broader teaching commons, it continued to evolve to the third level of public knowledge over time as we will explore further in this paper.

Stage One: Understanding Diversity
Our first research project, which began in September 2010, was aimed at understanding the nature of student diversity in the classroom, through a number of lenses.We could see the obvious differences, such as international students adjusting to a new language and culture, but we wanted to know as much as we could about diversity among our students across a number dimensions, including but not limited to academic preparedness, ethnicity, gender, and discipline of study.And we were interested in exploring how these differences might relate to learning outcomes (both perceived by students in survey response and measured in assessment and grading) within the course.The nature of this project was to understand what was happening in the classroom and the information was intended for our own use as instructors of the BUS100 class.We decided to use survey as our main instrument, along with some data from our institutional student information system.The course was designed to have strong pedagogical and assessment alignment with the intended learning outcomes.Assessment tools were continuously refined through the years as we learned through our research.We recognize there are many different methods to assess learning outcomes.
In developing our survey instrument, we used a combination of scales and questions derived from commonly used instruments we found through a literature review, as well as several instruments of our own design.These included Macaskill and Taylor's (2010) Autonomous Learning Scale, Duckworth's (2008) GRIT scale, Myers & Briggs' (2017) Myers-Briggs Type Indicator (MBTI), and Fleming's (2017) Learning Styles inventory.Instruments of our own design included students' perceptions of both experience and importance of different aspects of their learning, and specific questions aimed at understanding the students' experiences within and beyond the course.At this stage of the research, the only measured outcomes we used were course grades and overall GPA.First year GPA is not directly related to our course learning outcomes, but is the best overall benchmark we have for student foundations.
What we learned: We ran a regression to determine the strength of different variables in predicting the grades in the course, and were surprised at the strength (R2) of factors such as high school GPA (16%), writing skills (14%), and study habits (9%) in predicting grades and GPA.These results were exciting, as we could readily see things we thought we could do with them.Factors such as nationality (language and culture) and personality profile were also important predictors of grades, though they were not easily linked to actions within our control as instructors. 1hat we thought we learned: Our basic question in the research was about student diversity, and areas where there might be differences in experience and perceived outcome for certain groups or characteristics of students.We were quite pleased (perhaps prematurely) with the results, as student perceptions of value and learning generally did not differ among groups based on most measures of diversity.Analysis of variance showed no significant differences between any of the identifiable groupswith the exception of International students, who rated their experience more positively than their Canadian classmates.At this stage of the research, however, we were not paying sufficient attention to questions of validity and reliability of survey-based data, which we will return to below.Now we had the evidence we needed to change the world!We anticipated great opportunities to influence change based on the research results, but we would soon find out our sphere of influence was limited to our own efforts in our classes, at least for now.Based on these results, we placed our first emphasis on improving writing skills and study habits, where we developed early assessments designed to identify those who could benefit from interventions and support.An early assignment designed to assess writing skills (in particular, the students' organization and fluency) was used to flag students for supplemental advising.We also made changes to other major assessments based on the research results comparing grades by personality type.For example, "Judging" personalities, as determined by the MBTI tool, scored 8-10% higher than "Perceiving," which we interpreted as a possible grading bias.Therefore, we introduced more essay questions in exams, and allowed flexibility for student interpretation.Despite these changes, the gap in grades has remained.Research ethics presented a challenge for pedagogical change.For example, as part of the separation of teaching and grading from the research project, we were not permitted to view survey results on study habits until after the term was over.Therefore, results of the MacAskill and Taylor study habits surveys were not returned to students for their own formative development or used to help provide supplemental advising.
Reflecting on this stage of the journey, it is useful to think about the impetus, as SoTL projects are developed and undertaken, to avoid what Mills (1959) described as 'abstracted empiricism'disconnected studies that take on individual empirical questions without regard to the larger 'research imaginary.'The project used existing studies drawn from the literature to inform the research design, and while questions could be asked about the relative value of the components (for example, the work on learning styles, which has been discussed at length in the literature -see Pashler, McDaniel, Rohrer & Bjork, 2008), the project seeks to usefully adapt (with permission) existing instruments that allow more direct comparison with the existing literature in the field.Kember and Ginns (2012) provide an excellent starting point for exploring existing survey instruments that may help support new SoTL scholars' project development.Regardless of the instruments developed, used, or adapted, a useful reflective framework to assess the quality of SoTL projects is provided in Scholarship Assessed (Glassick, Huber & Maeroff, 1997).They articulate a framework for high quality scholarship that requires: 1) Clear goals (clear purpose, relevant and meaningful question); 2) Adequate preparation (through an understanding of the field); 3) Appropriate methodology (connected to the intended goals and questions of the project); 4) Significant results (that add consequentially to the field); 5) Effective presentation (that appropriately disseminates the results to the intended audience); and 6) Reflective critique (that critically evaluates the quality of the work).While our study does not cover all the breadth and depth of these SoTL leaders, we found their guidance valuable in shaping and improving our research.

Stage Two: High Impact Practices
From the launch of the BUS100 course, we were determined to employ innovative teaching practices and focus on our students.We tried different things in different class sections -just to see what works for us.Our biggest breakthrough, though, came when two of our team members attended the Mount Royal SoTL Symposium in 2010 where we met Carol Geary Schneider, who was leading a workshop on her work with George Kuh on High Impact Practices (2008) which increase rates of student retention and student engagement.We were encouraged to learn that we were already employing many of the tools and practices.We did not have a broad contextual understanding of why particular teaching practices were important or why they might be high impact, but we knew it was already working for us.
For stage two of the research, which began in September 2011, we set out to learn whether our teaching practices were working (i.e. were high impact) for our particularly diverse mix of students.We continued using and refining the research tools we developed for stage one of the project, as serving the needs of a diverse mix of students remained a key priority.At this stage of the research we added a component asking students to assess each of the major learning activities (which they could identify as their experiences in the class) in terms of effort, difficulty, and value.In addition, we asked them to assess their learning outcomes for university skills (general university skills, inquiry skills, writing skills, exam preparation skills, and group work skills) and business knowledge (environmental scanning, entrepreneurship, ethics, and management).
What we learned: The most important thing we learned at this stage of the project is that when one is adopting new teaching practices without the knowledge and experience to know how to make it work effectively, students will be brutally honest in communicating that it did not work out.Online tools for engagement were not received well initially, but we learned from the feedback loop and made improvements.The first-year seminars, into which we had devoted significant time and effort, did not show any meaningful impact on overall student learning.This nagging problem would have to be addressed at the next stage of our course development and of our research.For the initiatives which did not show initial success, we refined the practices, learned from experience and continued with both the innovations and the research process to understand the impact on student learning.Over time most of the high-impact practices received higher ratings from our students when asked about the value and impact that the practices had on their learning.
For another one of our key learning goals (inquiry skills), the student survey results did not match up to our assessed grades.Students self-rated strong learning of inquiry skills, yet our grading assessments showed weak performance by most of our students for these foundational skills.Again, these results helped identify an area which needed much more work in pedagogical development, and more work in refining the research instruments.Our final conclusion from this stage of the research was that, as classroom instructors, we had reached the limits of our capacity to influence the achievement of learning outcomes.To move beyond our somewhat limited successes to date, we would have to extend our sphere of influence and bring more players into our efforts to improve student learning outcomes.
Reflecting back on this stage of the project, our lessons learned were mixed.We put too much collective emphasis on student self-reported learning gains.Porter (2011) and Bowman (2011) have both shown that relying solely on student self-reported experiences or self-reported learning gains carries some risk.In their respective studies, they demonstrate (and conclude) that relying on student perceptions and self-reported behaviours or outcomes can lack validity and reliability, as students have been shown to have difficulty accurately reporting their behaviours and attitudes, and can even have problems answering basic factual questions accurately.The emphasis to this point of our research project on student self-reported outcomes (effort, difficulty, value, and impact) across a variety of measures forced us to reflect on what other measure of student learning (beyond grades) we might include in the project in order to provide assurances of the reliability and validity of the student responses to our surveys.
Additionally, one potentially valuable lesson for SoTL researchers to consider is that we often introduce something different in our teaching and assessment practices as part of a SoTL project, attempt to compare it to a control group on a onceonly basis, and publish the results.One practice SoTL researchers might consider is to plan for repeated periods of refining and measuring the impact of interventions before drawing any conclusions about effect on student learning or other possible research outcomes.

Stage 3: Foundations of Student Success
Stage 3 of the research coincides with our most recent efforts to expand our influence and build our team across the institution.In this stage of the research we focused more on factors and stakeholders beyond our classroom.More importantly, we realized a need to add SoTL expertise to our team.We recognized the weaknesses (and failures) of our project to-date, so we invited a more experienced SoTL scholar to help shape the continuation of the project.
For the purpose of our project, we identified five stakeholder groups who play key roles in student learning as we worked to set the foundations for student success.The stakeholders were: student access/admissions (recruitment and registrar), our instructor team, the faculty (department), institutional support services (library, international office, student success office), and the students themselves.Our focus at this stage, which began in September 2014, was to involve those stakeholders who might have a direct impact on student learning in our classes.Subsequently we would shift our focus to institutional policy and change processes at stage four of our project.
The major pedagogical shift we incorporated for the Foundations project at this stage was to incorporate support services from the library, international student support, and student success centre into our course programming.Library services developed online tools and other support materials, and scheduled seminars designed specifically to provide support for student inquiry skills in an environmental scanning project.Student success and international advising services were incorporated into the course structure to provide mandatory seminars (building on a model of intrusive advising) for students whose performance demonstrated that they were at-risk based on early assessments.Both of these major changes were developed based on research findings from the first two stages of the project, and on the challenges experienced during our earlier efforts at incorporating high impact practices into course.Both represent major departures from the traditional roles support services have played on our campus, though there are models of such partnerships in the literature (for example, see Frost et al., 2010).Stakeholders from these groups (both the library and student support services) are now integral members of our teaching team and key parts of the student experience in the BUS100 course.
As we progressed over time, the nature of the research continued to evolve.Figure 1 presents our research model as it has evolved up to Stage 3. It shows the complex array of information we are employing to understand our students, our teaching initiatives, student reaction to the teaching, and finally the achievement of learning outcomes (both measured through grades and perceived).While it may appear to be a complex set of relationships, we find that an understanding of the complexities is necessary to have a meaningful impact on foundations for student success.

Figure 1 Our Research Model -Foundations for Student Success
The research model represents the evolution of our research over time.Initially, we focused on the outer boxes -understanding diversity in student characteristics, and learning how those characteristics relate to both perceived and measured learning outcomes.Over time the research has shifted to the centre column -where we are doing applied research to learn how high impact practices, student engagement and belongingness, aliteracy (the finding that increasing numbers of capable readers are choosing not to read) and student accountability ultimately relate to learning outcomes.A new research instrument was introduced for this stage of the project to measure engagement, belongingness and self-confidence amongst the students (Yorke, 2014).We had learned from the literature that engagement is an important determinant of student retention and is a critical component of what makes high-impact practices 'high impact' (Kuh, 2008).Yorke's instrument enabled us to assess the students' perceptions about the importance of engagement, belongingness and self-confidence to the development of foundational university skills, the achievement of grades, and the overall perception of their learning experience (Yorke, 2014).As instructors we believe the most important contribution we make to a student's success is to engage them in a learning environment that challenges them to reach their academic potential, and to ensure they feel a sense of belongingness that enables their participation in an inclusive environment.This serves as personal motivation to continually seek out and experiment with innovative teaching practices.But we need to ask ourselves a question: Are our efforts at engaging the students leading to positive results?What we learned: The most important factor in determining student engagement is the approaches taken by a particular instructor leading a course section to create an engaging environment (regression R2 of 34%).The students' sense of belongingness follows a similar trend (29%).Engagement and belongingness, however, are not significant factors in determining grades regardless of the course section.In a stark contrast, student characteristics (gender, age, nationality 24%) and academic preparedness (high school average, study habits, writing skills 27%) are vitally important in determining grades, yet they are relatively unimportant in determining engagement and belongingness (under 5%).This result gave us some reassurance that our efforts as instructors are having an impact on the student experience of their learning environment.Grades for our course, however, are not closely related to engagement and belongingness.Instead, grades are determined primarily by student characteristics and academic preparedness before they enter BUS100.This aligns with other literature, for example, where the nationality of the students continued to remain a significant variable in predicting student success regardless of the teaching and learning intervention being planned or implemented in a given institutional context (Gebhart, 2012;Li, Chen & Duanmu, 2010).
The results, however, could act as a major demotivating factor for the team of instructors across BUS100.The student characteristics and academic preparedness at admission to university remain the biggest predictors of student success in this course, regardless of the learning experiences and environment created by the instructors.A growing portion of our students may not have sufficient skills (language, writing, reading, study skills) to succeed in university.To deal with this challenge, we more recently (September 2016) introduced a program of intrusive advising.All students whose early assessment grades (writing skills and first exam) fall below 60% are required to partake in workshops and seminars with our student success team, and their course capstone project options are limited to those which are focused on foundational skills.
The supplemental learning partnerships with our Student Success Centre and Library are resourceintensive interventions.Students receive individual and small group guidance in areas identified based on their early performance.Early results of these interventions show a significant positive impact on measured learning outcomes for those near the margins (early score average grades of 50-59%).Students who score lower than 50% on early assessments, however, were still as likely to fail the BUS100 class and to be placed on academic probation as they enter their second year.
What we learned: Students whose early performance was marginally below the standard were able to improve their performance and pass the course based on more intrusive interventions.Those who were significantly below the standard, however, did not pass the course regardless of any interventions we introduced over their first term.This was heartbreaking for the instructors.Based on the experiences from these previous semesters, we now know early in the term that a significant portion of our students will not pass, even with an intrusive advising model focused on supporting student success.We were initially expecting the majority of students in this category to be international students who were struggling with a new language and transition into a new culture.But about half of the students whose grades were below the thresholdeven with mandatory interventions -were Canadian domestic students.In the Fall 2017 semester, which was the second year of this intervention, the number of students flagged as at-risk, and with whom we engaged in an intrusive advising intervention, rose to over forty-five percent of the students enrolled in the BUS100 course.This presents a major problem for instructors, support staff and students -and will be a major focus of our attention in the next phase of the research.
The overall research model (see Figure 1 above) has served us well to-date in understanding what we need to know within our immediate sphere of influence over the course, but it has highlighted the stark reality that to enable student foundations for success requires interventions at the level of the institution, rather than the level of a single course.However, the research results have not yet served us well in influencing other stakeholders across the institution who have key roles to play in our students' learning outcomes.We will need to expand and shift our efforts in the next stage of our journey.The need to expand the scope of our work beyond the level of a single course aligns with recent literature about what matters most to the undergraduate experience.In The Undergraduate Experience, Felten et al. (2016) identify key areas of focus -learning, relationships, expectations, alignment, improvement, and leadership -that they argue matter most to improving the student experience and enabling student success.A focus on each of these factors at the level of an individual course is essential, but insufficient.Each require sustained involvement and engagement across the entire institution.As Astin (1984) has argued in his student development theory, in order to improve student learning we need to understand and influence student inputs (characteristics, academic preparedness) as well as the student environment (the curricular and co-curricular learning environments that the student experiences within higher education).

Stage 4: Partnerships for Student Success
As we explored the next stage of our collective journey, we were reminded of Elton's argument (2005) that the "the scholarship of teaching and learning is concerned not so much with doing things better but doing better things."Based on Stage 3 of our project (Foundations), the researchers recognize the limitations of our research at the level of a single course, as well as the potential benefits of broader institutional partnerships in improving student outcomes.Therefore, in addition to the primarily quantitative methods used in prior stages, the research will broaden in scope in two areas: we will expand the use of institutional data to enhance our understanding of the factors that influence student success, and we will interview key stakeholders across the institution to identify opportunities and barriers for instigating institutional change in a more systemic manner.
This stage of the research project is action and change oriented.The main goal will be to identify opportunities and barriers which may affect our collective ability to guide and support our students as they build their foundations for successin university and in life.We are expecting the research to identify recommendations for policies, pedagogical and curricular changes, and student success interventions to be taken across the institution.This project is ongoing through 2018, with plans to present findings and recommendations to university leadership in the coming year.
We have learned a great deal about our students and our pedagogical practices over the past decade, and are motivated to use that knowledge to enhance student success across the institution.We have determined that the incoming student characteristics and academic preparedness are important factors in determining student success.We have also found that the interventions possible at the micro scale of a single introductory class are not likely sufficient to impact the overall success trajectory of students at risk.

Conclusion
As we move into the final stage of this research project, our primary goal has shifted from understanding to influencing: Our aim is to convince the institution to investigate appropriate academic pathways that would support student success and help to channel students into appropriate student success programs.Different stakeholders all play a role in building student foundations for success, and we need to find ways to use our research results to engage each stakeholder group.With the realization that the educational context continues to change over time (pedagogical and assessment approaches, incoming student characteristics, etc.), we aim to transition our research efforts into ongoing routine measurement, monitoring and evaluation.We will continue to rely on our established partnerships within the institution to enhance student success, as well as our ongoing research partnership to continue to refine our research and evaluation processes.Ultimately, we believe that we (and here we mean a collective 'we' across higher education) MUST continue to focus on ensuring student success, retention and program completion.Indeed, "authentic motivation is…caught up in a struggle to do what is necessary and of value, not just for the organization nor just for oneself, but ultimately in the important interests of learners" (Grimmet and Neufeld, 1994, p. 5).
Our SoTL journey to-date has been fascinating and frustrating, but most importantly a rewarding experience.When we started the first stage of this project we did not know just how much we did not know about research methods and the SoTL commons.It has taken years to build our foundations as SoTL researchers, and to develop pedagogical strategies which have the greatest impact we, as instructors, are able to deliver for our students to build their foundations for success.But student learning goes far beyond what we are doing in our classrooms, and our research has shown us the need to expand our sphere of influence across the institution and beyond.Our most important research, and subsequent impact on student success, is just beginning now.