INCREASING CRITICAL THINKING IN PERSONAL DECISION MAKING- A CONTEXTUAL FRAMEWORK FOR COLLEGE STUDENTS

Statement of the Problem: The need for agricultural science students to graduate from institutions of higher education with strong critical thinking skills is expressed by both educators and employers to respond to the need for strong problem-solving, decision making, and analytic skills in the 21st century. To prepare students for the changing workforce evidence-based instructional practices, such as problem-based learning, need to be implemented into college courses to increase critical thinking skills. However, there are many barriers associated with implementing these teaching practices such as time barriers, large classes, and the challenge of measuring outputs related to critical thinking. The purpose of this dissertation was to understand how the addition of a contextual framework to an online nutrition module influences college students’ critical decision making (CDM). The aims of this paper are to 1) describe the role evidence-based instructional practices have on critical thinking skills in agricultural courses, 2) develop a guided scoring system to measure CDM, and 3) assess how two online modules using problem-based learning and scaffolding facilitate CDM when distributed in large introductory level classes. Methods: This research occurred in two phases, the first phase included the development of the online modules and testing of the reliability of the scoring system. Phase two was comprised of a randomizedcontrol trial where differences in CDM scores were evaluated between groups. The development of the two modules was guided by constructivism, using a problem-based learning and scaffolding approach. A contextual framework was created which included: framing the topic in the form of a question, organizational activities, and support in forming a decision. The guided scoring system was designed using a previously developed rubric to assess critical thinking when making a decision about food choices. The rubric was transformed into a guided scoring system to assess whether students 1) made a decision, 2) used evidence to support their decision, and 3) addressed the opposing point of view. For phase two, students were individually randomized into the intervention group with a contextual framework or the control group without the framework. The modules focused on two topics related to environmentally conscious eating (protein sources and organic foods). Groups were exposed to the same instruction, shown two identical videos, and asked to make a decision about each issue. Summary of Results: Based on phase one, the results showed that the scoring system was reliable, and the modules were successful in promoting CDM. Overall, the CDM framework was found to be successful at encouraging decision making and using evidence to support the decision. The changes in score were captured through the guided scoring system. In phase two, the results give evidence that providing a contextual framework helps students utilize CDM skills. Future interventions should consider using the CDM framework when aiming to increase CDM.


INTRODUCTION
To address the changing environment of the workforce and technology advancements, researchers have become increasingly interested in how to develop stronger critical thinking skills in undergraduate agriculture science curricula (Easterly, Warner, Myers, Lamm, & Telg, 2017). Agriculture science programs are defined in this review as majors that include, but are not limited to, animal science, human nutrition and food science, crop production and management, sustainable agriculture, and environmental studies. Higher education curricula need to be designed to encourage the use of critical thinking skills within these majors because it will better prepare students for the workforce by developing their skills in problem-solving, strategic planning, and communicating complex issues (Easterly et al., 2017;Huber & Kuncel, 2016). Critical thinking outcomes have been extensively studied in other programs such as nursing and medical education programs , but less attention has been paid to agricultural science curricula (Burris & Garton, 2007).

PURPOSE OF THIS STUDY & OBJECTIVES
The purpose of this literature review is to describe the research that has been conducted on learning theories aimed at increasing critical thinking in higher education agriculture science classrooms. 1) Describe the role teaching methods have on critical thinking skills in agricultural classes.
2) Determine successful strategies and barriers found when implementing strategies to increase critical thinking skills.

st -Century Skills
The skills needed to be successful in today's world include critical thinking, collaboration, and problem-solving leading to informed decision making. Due to the rapid changes in technology and environmental circumstances, students are being trained for jobs that do not yet exist, as they will be living and working well into the 2060s and 2070s (Halpern, 1998). Having the ability to think, adapt and apply concepts in a changing environment is essential for students to be successful in their future careers. This requires students to be lifelong learners and develop metacognition (Livingston, 2003), which is defined as learning how to learn and seek out information rather than relying on recall or memorization.
Higher order thinking skills give students the ability to work through problems, develop innovative solutions, and the tools to be lifelong learners. Agricultural science classes should naturally require these skills in students, especially when teaching topics focused on environmental issues such as climate change, the food system, and sustainability. These topics require critical evaluation because there is not a one size fits all resolution, and differing viewpoints for solutions can be easily debated with evidence.

Higher Order Thinking Skills and Bloom's Taxonomy
Higher order thinking skills are most popularly defined by Bloom's Taxonomy which divides learning into lower order thinking and higher order thinking (Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956) (Figure 1). Bloom et al. (1956) classified learning into six categories that are made up of three lower order thinking (i.e., note taking and memorization) and three higher order thinking (i.e., interpret, analyze, and assess the information). Higher order thinking involves creative thinking, problem solving, questioning, and systematic thinking which help people to succeed when facing unfamiliar situations . Critical thinking is often used as an indicator of higher order thinking skills because it can be measured through validated and reliable instruments (Miri, David, & Uri, 2007).

Critical Thinking
While the importance of critical thinking is recognized within higher education, there is variability in its definition. Researchers have identified specific skills to define and assess critical thinking competencies which include: analyzing, applying standards, discriminating, information seeking, logical reasoning, predicting and transforming knowledge , all of which are important for college graduates to possess so that they may solve problems more effectively (Snyder & Snyder, 2008) (Table 1). These skills describe a person who is actively engaged in analyzing and evaluating information for meaning and solutions to problems . A strong critical thinker is also able to evaluate evidence and utilize purposeful and introspective approaches when faced with a problem or question . Recognizing similarities and differences among situations Analyzing Breaking down a problem into its different parts to discover their nature, function, and relationships Transforming Knowledge Changing or converting the condition, nature, form, or function of concepts among contexts Predicting Envisioning a plan and its consequences Applying Standards Judging according to established personal, professional, or social rules or criteria Logical Reasoning Drawing inferences or conclusions that are supported in or justified by evidence critical thinking characteristics predicted student profiles of "risk taking" and "lost in translation" (Franco, Costa, & Almeida, 2017). Similar results were found by , who surveyed community adults (n= 50), community college students (n=35), and state university students (n=46). The results showed that those who had higher critical thinking skills reported significantly fewer negative life decisions, adding to the argument that critical thinking may lead to better decision making. These critical thinking skills need to be fostered so that students have a natural inclination, or disposition, to consider facts, recognize gaps within the evidence, and evaluate all the choices when making decisions .

Disposition Related to Critical Thinking
Critical thinking disposition is defined as an individual's "internal motivation to use critical thinking skills" (Pascarella, Terenzini, & Feldman, 2005). For example, students who are taught critical thinking skills may naturally choose to not use those skills because of a lack of internal motivation (C. Friedel et al., 2008;Friedel, Irani, Rhoades, Fuhrman, & Gallo, 2008;Stupnisky, Renaud, Daniels, Haynes, & Perry, 2008). Within college students, higher critical thinking disposition has been linked to inquisitive behavior, openmindedness, making unwarranted conclusions cautiously, and carefully evaluating information for credibility (Pithers & Soden, 2000). Critical thinking disposition has also been found to increase after students are exposed to teaching methods that utilize critical thinking (C. Friedel et al., 2008;Tishman & Andrade, 1996), but studies have also shown that if students do not possess the skills to critically think then overtime that inclination towards critical thinking may decrease (Stupnisky et al., 2008). This is why incorporating critical thinking into college courses is imperative because it can lead to both an increase in skill and disposition, but critical thinking is a difficult construct to measure .

Critical Thinking Assessment
The most popular tools to measure critical thinking include the Watson and Glaser Test, the Cornell Critical Thinking Test, the California Critical Thinking Skills Test , the Halpern Critical Thinking Assessment, The Critical Thinking Assessment Tool (CAT), and the California Critical Thinking Disposition Inventory (Table 2). While these are validated instruments that measure critical thinking, they are lengthy and time consuming for the participant which prevents realistic use in the classroom.
For example, open-ended tools, where participants write out a response to a situation, have proven to be reliable, but take time to administer and increase the burden on the researcher to score .
The central issue in measuring critical thinking is the type of memory the instrument uses. For example, recall-based memory (short answer, essay) items are more difficult to measure, may be at risk for reliability errors, and have a high cost to administer. On the other hand, while recognition memory (multiple choice, ranking) instruments are less expensive to score, they allow the participant to guess answers and may have weaker validity . Instruments that measure both types of memory tend to be long and impose a substantial subject burden which hinders their classroom use . Additionally, with continued research, many of the critical thinking instruments have found to be inconsistent in terms of validity and reliability  and may not be sensitive enough to measure indicators of the use of critical thinking in specific courses.  Bers, McGowan, & Rubin, 1996;Giancarlo & Facione, 2001;Klein, Benjamin, Shavelson, & Bolus, 2007;Pascarella et al., 1999;Saavedra & Saavedra, 2011;Saucier, 1995), no increase, or a decrease in critical thinking skills during the college years (Blaich & Wise, 2010;Phan, 2011;Pithers & Soden, 2000;Schendel, 2015). It is suggested that to facilitate critical thinking, instructors will have to move away from instructor-centered teaching methods, and towards learner-centered curriculum (Brown, 2003;, which allows students to "construct knowledge by gathering and synthesizing information through inquiry, communication, critical thinking and problem solving" (Huba & Freed, 2000). Teaching approaches that have shown to increase critical thinking skills include: experiential learning , case studies (Popil, 2011), and writing activities (Tsui, 2002), where students are participating in active learning and building on previous knowledge; all of which implement the constructivism theory.

Constructivism Theory of Learning
As a theory of learning, constructivism states that students learn by connecting concepts to previous knowledge and experiences, and build knowledge based on what they already know rather than starting with a blank slate . Three main points that summarize the important components of constructivism include: 1) knowledge comes from interactions within the environment, 2) a goal for learning encourages engagement, and 3) knowledge evolves based on social negotiations . The constructs of social constructivism require the learner to play an active role in the learning process. However, the learning process should be structured to help the learner move towards independence while completing complex tasks . This structure, called scaffolding, can be added or removed to help students complete complex tasks on their own. The instructional principles of constructivism as a learning theory are summarized in Table 3. Provide support for reflection of content and learning process Allow the learner to analyze and make judgments related to the learning experience 1. Savery and Duffye, 1996 Problem based learning The constructivism theory can be implemented into classes using problem-based learning (PBL). The PBL model is a student-centered approach to learning in which students are presented with an authentic scenario related to a classroom topic and are tasked with generating solutions . This model implements the constructivism theory because it creates a learning environment that requires the student to play an active role in learning and lead their own investigation into facts (Hendry, Frommer, & Walker, 1999). This type of teaching method leads to an increase in students' critical thinking skills by requiring problem-solving, evaluation of facts, weighing evidence, and ultimately making decisions based on what is known (Dochy, Segers, Van den Bossche, & Gijbels, 2003;Kek & Huijser, 2011). Problem-based learning interventions have shown to be successful in increasing critical thinking skills while achieving the same level of learning (Chapman, 2001).
The ideal method for utilizing PBL is through reasoning, where the student is confronted with competing logics or points of view . Social-problems, using real-world issues, is the ideal topic because they are multi-logical, controversial, and lack a clear or "correct" solution . This is used to engage the students' critical thinking skills and help them become motivated to solve the problem (Kek & Huijser, 2011). Problem-based learning lends itself nicely to agricultural studies because it allows learners to participate in critical thinking by debating solutions to topics and evaluate multiple perspectives (Wals & Jickling, 2002). Despite its easy application to agricultural science, PBL is often not utilized in introductory level agriculture science courses. However, it has been implemented successfully in medical and nursing programs.

PBL and Medical School
Problem-based learning originated within medical school curricula around the year 1960 and since then many other medical programs have made the move to incorporate this teaching style . , conducted a systematic review looking at the effect of PBL curricula during medical school and physician competencies later in life.
Thirteen articles were included that fit their inclusion criteria, all of which include an intervention group (PBL) and a control group, had a PBL curriculum for at least two years, and assessed physician competencies one year to twenty years after program completion.
Overall, the studies reviewed showed that the physicians in the PBL had higher scores for social skills, which included, "teamwork skills, appreciation of social and emotional aspects of health care, appreciation of legal and ethical aspects of healthcare, and appropriate attitudes toward personal health and well-being", and there was moderate support for greater skills in communication and interpersonal interactions. Ultimately, the reviewed articles supported the use of PBL in medical schools' curricula because of the positive associations found between PBL and increased social/cognitive skills. Nursing programs have also realized the importance of including PBL in their approach to learning and have researched its impact on critical thinking. , conducted a systematic literature review focusing on the use of PBL in nursing curricula and its impact on critical thinking skills. Ten studies fit their criteria, and the results showed that incorporating PBL into the different nursing curricula had a positive correlation with certain critical thinking skills, such as, analyzing problems, considering other solutions to problems, and recognizing the need for more information when compared to a lecture style class. However, the studies did not show a significant increase in overall critical thinking scores, which the authors attributed to a lack of validity in the tools used to measure critical thinking and the lack of large randomized control trials.

PBL and Nursing
A more recent meta-analysis conducted by , also investigated the relationship between PBL and critical thinking in nursing students. This meta-analysis included eight articles that met the criteria of being a randomized-controlled trial, reported critical thinking score as the outcome, and used PBL for the intervention strategy. Problembased learning was implemented by framing the learning within authentic problems and then compared to traditional lecture style groups. The results from the analysis found that in the pooled sample of students (N=910) there was a significant increase in critical thinking score using PBL when compared to the lecture-based group. The long-term outcomes between PBL and critical thinking are still being investigated, but there is evidence that PBL can increase specific skills related to critical thinking such as problem-solving ability, investigating facts, and informed decision making .

Agricultural Science Education
Leaders within agricultural science education have identified the need to implement "intentional methodologies" to prepare students to solve complex problems facing society (Andenoro, Baker, Stedman, & Weeks, 2016). The American Association for Agriculture Extension's (AAAE) strategic research plan for 2010-2020 highlights the need to 1) investigate "what methods, models, and programs are effective in preparing people to solve complex, interdisciplinary problems (e.g., climate change, food security, sustainability, water conservation, etc.)" and 2) develop evaluation methods to measure the effectiveness of educational programs to prepare students for careers in agriculture and natural resources . These two areas of research are ranked as the top two priorities of the AAAE in order to prepare students to be critical decision makers when faced with the complexities of a changing environment and growing population (Andenoro et al.,

2016).
A widely implemented instructional practice within agriculture science is experiential learning, most commonly through Supervised Agriculture Experience (SAE) programs or "authentic" learning experiences (Knobloch, 2003). Experiential learning falls under the social constructivism and PBL approach to learning because it takes into account that students often draw on their own daily experiences when solving problems, for example, interactions that they have had with friends, family, and situations embedded in their environment (Mughal & Zafar, 2011) and is a learning experience where students are developing solutions to problems through a hands-on practice (Pennington, Calico, Edgar, Edgar, & Johnson, 2015). While experiential learning, using PBL and a constructivism approach, is theorized to encourage deeper learning of content, the relationship between it and critical thinking has not been investigated.

Gap related to Agricultural Science Education
While there is extensive research and reviews within medical schools and nursing curricula implementing specific teaching strategies to increase critical thinking, there have been no reviews focusing on agricultural science programs instructional practices aimed at increasing critical thinking. Which raises the question, what teaching methods have been implemented to facilitate critical thinking in higher education agriculture science courses, how is critical thinking being measured as an outcome, and what barriers are instructors facing?

METHODS AND PROCEDURES
To identify relevant studies an electronic search using the databases PubMed and EBSCOhost was conducted for articles that focused on increasing critical thinking abilities in undergraduate agricultural science curricula. The search was limited from January 2000 to December 2017 to include current literature. The key words "critical thinking", "agricultural science" or "science", and "undergraduate" were used to conduct the search. An additional search was conducted using the terms "online module" and "critical thinking" to explore online methodologies for teaching and its impact on critical thinking. The studies were included in the review if they met the following inclusion criteria: (1) used a specific teaching approach as an intervention, (2) included undergraduate agricultural science classes, (3) evaluated critical thinking as an outcome. A total of 33 articles were found that met the initial search criteria, 12 of those articles were excluded based on not including the target population (non-higher education), were not in English, or did not identify a specific learning theory to assess the influence on critical thinking or related outcome. This resulted in 14 articles that were included in the review of critical thinking and agricultural science classes (Table 4) and seven articles focusing on increasing critical thinking skills using online delivery. Overall, there is limited research that explores the role of critical thinking in agriculture courses when examining scope and rigorousness of study design . Mixed results have also been found when comparing agriculture science students' critical thinking skills to national norms. Perry, Retallick and Paulsen (2014), found that agriculture students' (n=75) critical thinking skills in problem-solving were comparable to national averages. These results were similar to Friedel et al. (2008a), who found that their sample of agriculture science students (n=108) also scored similarly to national norms in critical thinking disposition and problem-solving level. However, previous research conducted by , measured students' (n=174) critical thinking disposition and found that about one-third of the students (30.5%) had a weak disposition towards critical thinking and only about 2% were found to have a high disposition. Alternatively, Rhoades et al. (2009) found that across four universities students majoring within agriculture science (n=178) had a moderately-high critical thinking disposition but their critical thinking disposition score was found to be significantly lower when compared to students who were non-agriculture science majors. Findings from these studies suggest that instruction needs to be intentionally designed to foster these skills and encourage students to use critical thinking (Gunn, Grigg, & Pomahac, 2008). This requires a shift from teacher-centered instruction to student-centered learning opportunities. Research has shown that with professional development faculty begin to adapt teaching styles that foster critical thinking, which is often not taught during their doctoral and post-doctoral work (Baiduc, Linsenmeier, & Ruggeri, 2016).

Faculty Professional Development for Instruction
A meta-analysis conducted by , found that instructors with advanced training in teaching critical thinking skills had the greatest improvement in critical thinking skills of their students, however, this was not discipline specific. Focusing on agricultural studies, a study conducted by , examined the impact of a yearlong faculty training course on how to explicitly teach critical thinking in agriculture classes. This sample included 12 faculty members who taught 14 agriculture science courses. The faculty was trained on how to overtly teach and assess critical thinking within the classroom and how to develop stronger critical thinking disposition within students.
Faculty also participated in monthly meetings in which they shared teaching strategies, course revisions, and were provided with feedback on their teaching plans. The results showed that from pre-semester to post-semester, students in 12 of the 14 classes increased their critical thinking disposition score, which supports the conclusion that professional development for faculty can increase students' critical thinking disposition within a semester. However, research is lacking about which teaching strategies are being implemented in agriculture sciences classes that increase critical thinking skills. Lohse et al. 2003, implemented PBL into a nutrition across the lifespan class, where students' (n=32) critical thinking (evaluated by reflective thinking write-ups) results were compared between a lecture style class and a PBL class. The students in the PBL class reported greater enjoyment of their learning environment, but the results showed that there were no significant differences between groups on critical thinking, however this may have been a result of the small sample size or a result of the way critical thinking was measured through reflective write-ups.
A qualitative study conducted by  investigated how faculty at two universities assessed critical thinking in their classes. The sample included tenured faculty (n=18) who taught general education classes within the fields of either humanities, natural sciences, or social sciences. Findings showed that while faculty reported using specific coursework for teaching critical thinking and implicit assessment of critical thinking skills (e.g., "I know it when I see it"), they did not explicitly measure critical thinking using validated or established tools specifically for critical thinking assessment.
This is problematic because, as the authors conclude, "there is no way to know the efficacy of their efforts to develop critical thinking in students" . The article showcased the importance of using unambiguous tools to teach and measure critical thinking. , used a standardized critical thinking assessment, the Cornell Critical Thinking Test (CCTT), to measure critical thinking at baseline and post semester in a food science and human nutrition course. The instructors designed the course using PBL activities to determine the effects on students' critical thinking skills. This study was conducted over an 8-year period (2001-2008) and found that in only two of the years (2002 and 2004) there was a significant increase in critical thinking scores, but no significant increases in the pooled sample (n=154) over the eight years. Similarly, a study conducted by , measured critical thinking at pre and post, using the Critical Thinking Assessment Test (CAT), in a senior capstone farm management course using a validated critical thinking assessment. The course was designed using PBL, by providing students with an opportunity to implement previous knowledge while managing a farm and required the use of teamwork, problem-solving, and decision making. The results showed that there were no significant changes in the larger construct of critical thinking, but that students scored significantly higher in terms of problem-solving when compared to national norms.
Both studies  noted that based on assignments and course content students collectively began to think about how they worked through problems and developed stronger metacognition skills, which may not have been captured within the constructs of the CCTT or the CAT. This acknowledges the fact that a standardized test may miss gains in critical thinking abilities, which brings to question whether a standardized critical thinking assessment is the best way to capture changes in critical thinking skills.
Alternatively, critical thinking has also been measured using qualitative methods. A study conducted by Harman et al. (2015), measured critical thinking through written responses (n=426) and focus groups (n=3) with a total of 85 students to investigate how a method of PBL (case-based learning) impacted critical thinking outcomes. This study was conducted in two upper-level undergraduate nutrition education courses where case studies where implemented to guide students through PBL and heighten critical thinking skills by solving problems that students may see in a professional setting. Themes that emerged from the qualitative analysis included increases in problem-solving abilities and a greater number of students recognizing the implications towards future professional practice. However, while the themes that emerged from the study were related to high-order thinking, critical thinking skills were not explicitly measured.
A similar study conducted by Heinrich et al. (2015), used four different sites in a sustainability course for experiential learning to increase critical thinking skills, but instead of analyzing open-ended responses using emerging themes, the researchers scored written responses with a rubric. The results showed that when experiential sites explicitly used critical thinking as a learning outcome there were higher scores for skills related to critical thinking, such as using evidence to explain reasoning and questioning thinking. Results across the four sites were not comparable, though, because each site used a different openended activity to score critical thinking along with different experiential teaching methods, which makes it difficult to assess the impact PBL had on critical thinking skills over time.
Additionally, the studies reviewed thus far were implemented within semester long courses with relatively small class sizes. Based on the research thus far, there is a rather small reach and unrealistic scalability of implementing critical thinking activities in large college courses.
Contributing to this point, a study conducted by , integrated a competing narratives approach, which is a form of PBL, into a large (n=209) introductory natural resource conservation course. The approach consisted of overtly teaching critical thinking and assigning three writing assignments where students compared and contrasted conflicting texts about climate change (evaluated differences). Pre-post critical thinking ability was measured using the Critical Thinking Assessment Test (CAT) where students answered 15 short essays, which were then scored. However, due to the time burden of scoring the responses, only 37 pre-post matched pairs were scored. Students showed a significant increase in their critical thinking skills pre-post and had scores significantly greater than national means. Overall, using competing narratives was found to be successful in a large introductory course, and the researchers attributed this success to 1) choosing a relevant/controversial topic, 2) connecting multiple disciplines, and 3) discussing misconceptions related to the topic. However, the time burden of scoring the CAT makes it challenging to measure critical thinking in a large class setting and without a comparative group accrediting all of the results to the competing narratives approach is not generalizable. results between an explicitly taught course using overt teaching methods of critical thinking (n=20) to a nonequivalent control group (n=30). The critical thinking course used methods to overtly teach critical thinking, which included teaching about skills involved with critical thinking (e.g., interpretation, analysis, evaluation, inference, explanation, and selfregulation) and then asked students to apply those skills throughout their class activities.
Critical thinking was measured using the University of Florida Critical Thinking Skills Test.
The results showed that the critical thinking group had a higher critical thinking score at post compared to the nonequivalent control group but with a small sample size and a nonrandomized sample it is difficult to generalize to different samples and course content.
Researchers have overcome the barrier of in class administration of critical thinking activities and assessment by implementing online delivery methods (Table 5). A systematic review conducted by Jin et al. (2014), found that within the articles reviewed (n=28) there were positive effects on learning when PBL approaches were implemented using out of class technology. There were three main points highlighted in the review: 1) implementing PBL online gave students an opportunity to be exposed to information using diverse modalities including case studies, videos, virtual patients, and discussions; 2) instructors were able to design courses that used both PBL approaches and explicitly teach information related to the topic area; 3) by using methods of technology that guided learning, students were able to be exposed to learning activities with built in scaffolding that guided them in forming structured responses. The three points described relate back to the social constructivism model. This allows students to form knowledge and learn through a process with built in supports to move towards independence in their problem-solving abilities. Online learning also allows for a personalized approach to learning, which may be difficult to implement in large introductory classes. Moreover, online learning provides flexibility both in time constraints and environment, for students to access the course material. However, it is important to consider the impact online learning can have on critical thinking. Critical thinking skills increased significantly more when students were exposed to PBL online learning modules compared to lecture-based modules Re-test for reliability/validity Online conditions may lead to greater development of critical thinking skills when compared to an in-person discussion. Guiller et al. (2006), compared differences in critical thinking indicators between transcripts of an online discussion condition to an in-person condition when discussing scientific articles. The results showed that during the online condition students expressed more points of view on the topic and empirical evidence to support their decision when compared to the in-person condition. Richardson and Ice (2010), further investigated the impact of online discussion questions on critical thinking by testing three teaching methods: debate, case-based discussion, and open-ended discussion. The researchers hypothesized that students' critical thinking levels would differ based on which of the three teaching strategies were implemented. To measure critical thinking, the Practical Inquiry Model (PIM) was used which scores open responses on the process of thinking rather than on a specific thinking outcome. PIM has 4 phases: triggering, exploration, integration, and resolution, which account for different levels of thinking. All students (n=47) participated in the three methods of learning and were assessed using qualitative data in the form of online discussions (n=2516) and quantitative data that surveyed their experiences with the teaching methods and online discussions. A high percentage (47%) of the students reported that they preferred the open-ended activity the most (over debate and case-based) because 1) they had more opportunity to express their opinion and 2) there were no right or wrong answers. Based on the qualitative data that was used to assess critical thinking, there were no significant differences found between teaching strategies on the four phases of PIM. However, the results showed that all three teaching methods were successful at facilitating critical thinking and many of the posts were categorized at reaching the integration stage, which indicates that students were developing solutions and connecting ideas when participating in the discussion.
Beyond using online discussions, a fully delivered online learning module was feasible for teaching critical thinking skills. A pilot study, conducted by Carmichael and Farrell (2012), developed online modules about critical thinking based on interviews with both students and professors. The learning modules used interactive activities and multimedia to teach students about critical thinking and provided additional resources relating to different majors. The pilot study investigated the use of the website and the usefulness from the student perspective. Results of the study indicated that students were using the site and reported that the learning modules were helpful in understanding critical thinking skills. However, there were no tests measuring students' critical thinking skills to evaluate the impact the learning modules had on student outcomes. Sendag and Odabasi (2009), did find that critical thinking skills increased significantly more when students were exposed to PBL online modules compared to lecturebased modules. Students were randomly placed in either the PBL group (n=20) or the lecture-based group (n=20). Critical thinking was assessed at baseline and post using the Watson-Glaser critical thinking test. At three times during the semester, the PBL group was exposed to problem situations and were instructed to develop solutions to the problems and discuss questions using an online forum. At the end of the semester there was no difference between groups in terms of learning outcomes (i.e., test scores), but the PBL group scored significantly higher on critical thinking skills compared to the lecture-based group, indicating that PBL helped students develop stronger critical thinking skills by requiring them to think in terms of generating solutions, evaluating the research, and forming a decision. Similar interventions with larger samples should be implemented to test for reliability and validity of an online PBL intervention to increase critical thinking skills.

CONCLUSIONS
The objective of this review was to 1) identify the impact evidence-based instructional practice, such as PBL, have on critical thinking, 2) to identify how critical thinking skills are being measured, and 3) identify barriers that are faced when implementing evidence-based instruction within agricultural science curricula. As demonstrated in this review, there is limited research that explores the impact of designing and implementing evidence based instructional practices within agricultural studies to make conclusions on how to foster and measure critical thinking skills on a large-scale basis. The limitations in the literature involve both the scope and lack of intervention-control trials of research, along with limitations due to small sample sizes and measurement of critical thinking skills. However, there is supporting evidence to merit further investigation that using PBL within agriculture studies encourages the use of critical thinking skills.
Problem-based learning has shown to be successful at encouraging students to use critical thinking skills and found to result in greater critical thinking skills when compared to traditional lecture centered teaching methods (Şendağ & Ferhan Odabaşı, 2009). The literature reviewed has supported the positive relationship between PBL and critical thinking skills across in-person instruction and online methodologies (Harman et al., 2015;Heinrich et al., 2015;Şendağ & Ferhan Odabaşı, 2009;. However, more research is needed to successfully identify how to incorporate these learning strategies and measure the outcomes in larger introductory courses. Many of the limitations within the studies were due to a lack of measuring critical thinking skills because of research burden (Harman et al., 2015;Heinrich et al., 2015;, and this limitation was reinforced by small sample sizes (Lohse et al., 2003;. The reviewed research highlighted the difficulty of measuring critical thinking and the benefits and limitations with using either multiple choice or open answer responses . Alternatively, critical thinking disposition was used as an outcome measure as an indicator of critical thinking skills Rhoades et al., 2009;, along with rubrics that were specifically designed to measure components of critical thinking (C. Friedel et al., 2008;Harman et al., 2015;Heinrich et al., 2015;Lohse et al., 2003). This underscores the need to develop rigorous studies that measure outcomes and contain comparison groups to identify best practices in measuring critical thinking.
Barriers to implementing PBL has been provided in this review. The majority of the studies conducted thus far used a cross-sectional or single group study design, which inhibits comparison of outcomes between groups or between teaching styles. Also, the studies were semester long courses and discipline specific, which raises the question if short-term interventions could have similar impacts on critical thinking ability. These factors (e.g., no outcome measure, one-group study design, and small sample sizes) make it difficult to assess the true impact PBL has on critical thinking skills within the agricultural science curriculum.

DISCUSSION AND IMPLICATIONS
The need for 21 st -century skills is necessary now more than ever. As students advance through their higher education courses, they are being trained for jobs that may not yet exist in today's job market (Perry, Michael S. Retallick, & Paulsen, 2014). Students must be prepared to be lifelong learners and develop higher order thinking skills, such as critical thinking, that will prepare them to be successful in their future careers (Easterly et al., 2017).
The use of PBL has found to be successful in developing critical thinking skills in other areas of study (medical and nursing programs) and lends itself nicely to be applied into agricultural science curricula. There is limited evidence to draw conclusions about the effects PBL has on gains in knowledge and achievement over lecture-based classes.
However, there is evidence that PBL interventions are successful in increasing critical thinking skills while achieving the same level of learning (Chapman, 2001).
Problem based learning within agriculture science classes has been implemented in multiple ways, however, using PBL effectively may require professors to be explicitly taught and instructed about how to incorporate this teaching method into their classes (Lund & Stains, 2015). While professors are experts in the classes they instruct, the lack of professional development in teaching methods can result in challenges, such as time constraints, few incentives and low self-efficacy in teaching in terms of critical thinking (Stieha, Shadle, & Paterson, 2017). Assisting professors in implementing evidence-based instructional practices, such as PBL, could be done by developing frameworks or outlines that can be easily manipulated to fit multiple topics. This would provide enough guidance to facilitate critical thinking instruction within a class, while allowing for individual tailoring of the content by the professor.
Additional implications for this area of research may include new methods to implement PBL and increase critical thinking. Researchers have demonstrated that online learning modules can be successful in facilitating learning within large classes, which can be implemented into courses when faced with time constraints. For instance, a study conducted by Corrigan et al. (2008), developed online modules to aid in clinical decision making in a large (n=116) undergraduate medical course. Results showed that after participating in the supplemental out-of-class online activities, students' scores on their final clinical exam were significantly higher compared to previous years that were taught using in class lecture-based instruction. Using online modules to test developed frameworks for critical thinking would also be a feasible method of engaging large amounts of students which would allow for assessment of the effectiveness of evidence-based instruction on critical thinking skills in large classes. Furthermore, students report that they would prefer to participate in an online learning module where they have the ability to explore concepts and information at their own pace, time, and location rather than a face-to-face discussion where they have limited access to resources . Implementing PBL where students are connecting learning to real-world application has shown to lead to an increase in critical thinking.
Furthermore, using PBL methods that require students to solve and analyze complex problems have shown to better prepare students for future careers.

PRACTICE
One goal of higher education is to prepare students to be critical thinkers, which in turn produces graduates who are capable of gathering, assessing, and interpreting information to form creative solutions to multi-faceted problems. As the number of young adults attending college continues to rise there is a need for wide dissemination of teaching strategies that will foster critical thinking skills. Future studies should focus on overcoming gaps that were highlighted in this review such as testing critical thinking interventions in larger samples, incorporating a comparison group into the design of interventions to better evaluate the impact of teaching methods on critical thinking outcomes, and measure outcomes that capture critical thinking skills using either qualitative or quantitative tools.
Using technology as an out of class activity may allow for the testing of instructional frameworks in larger samples and assessment of feasibility by allowing students flexibility in participation. Technology allows content to be tailored to more personalized learning and structure that may be unrealistic in large introductory level classes because of space and time limitations. However, while previous research has shown that using technology for supplemental learning in classes can be beneficial for students, the content of the online coursework should be developed to encourage motivation from the student to learn and engage in the content.
Another area where research can expand upon is how to measure critical thinking or constructs related to critical thinking, within a course that has a large number of students enrolled. Many of the articles included in this review cited critical thinking as an overall objective, but failed to overtly measure it, making it difficult to form inferences and conclusions based on the research. Ideally the use of validated and tested instruments would provide evidence of the use of critical thinking skills, however existing instruments may not be feasible to administer in large classes because of subject and research burden. The literature reviewed offered alternative methods of capturing critical thinking skills using rubrics designed to assess the use of critical thinking within class activities and participation.
Future research should investigate the practicality of implementing rubrics to measure critical thinking skills within large classes.
Lastly, further research should focus on how specific teaching methods within agricultural science impacts critical thinking and outcomes later in life. Based on the work reviewed, there is clear importance that teaching faculty needs to continue to design courses with the intention to facilitate critical thinking. The facilitators of critical thinking include motivating the student to participate in learning, aiding them in organizing their information or course content, and providing support to help construct ideas and learning related to the topic area while connecting the topic to real world authentic situations. Providing professors with frameworks, or contextual guides, to implement these evidence-based instructional practices may help to overcome barriers (e.g., time constraints, awareness, feasibility in large classes) that are expressed by instructors within the literature.
There is a large body of research related to defining critical thinking and researchers have highlighted the role of critical thinking within higher education, but there is a lack of interventions that use rigorous design to isolate differences in critical thinking outcomes when students are exposed to different teaching methods. Further investigation should focus on how critical thinking can be increased in higher education agricultural science courses and methods of implementation using large scale designs. As there are more calls within higher education to reform curricula to prepare students for academic and career success, agriculture science education must also evaluate the effectiveness of teaching methods to facilitate critical thinking. Furthermore, questions remain about how increasing critical thinking through coursework influences students as they move on into the workplace and how being a stronger critical thinker can influence their decision-making ability as citizens. , E., & Farrell, H. (2012)

Introduction
Twenty-first century skills that are necessary to be successful in today's workforce include collaboration, communication, critical thinking, and problem-solving. The ability to think, adapt and apply concepts in a changing environment is essential for students to be successful in their future careers. This requires students to be lifelong learners, to develop metacognition, which involves reflective practice of learning how to learn (McGuire & McGuire, 2015) and to seek out information critically rather than relying on recall or memorization. Students will be better prepared for the changing workforce if these 21 st century skills are fostered. One way of doing this is by developing higher order thinking skills, which gives students the ability to work through problems and develop innovative solutions.
Students often spend their first two years of higher education in large lecture and introductory classes where higher order thinking skills are not being developed (Gasiewski, Eagan, Garcia, Hurtado, & Chang, 2012). Many of these introductory courses focus on lower-level thinking which includes the memorization of facts, without application to novel problems or situations . Research shows that developing higher order thinking skills requires repeated exposure because skills develop over time. If these skills began developing earlier on in college, then students may be better prepared for higher level and capstone courses which require more in-depth knowledge application .
Higher order thinking is most popularly defined by Bloom's Taxonomy which splits learning into lower order thinking (i.e., recall and memorization) and higher order thinking (i.e., interpret, analyze, and assess the information) (Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956). Higher order thinking involves a multitude of thinking skills (creative thinking, problem-solving, questioning, systematic thinking) which can be used to solve problems when faced with unfamiliar situations . Critical thinking (CT) skills are often used as an indicator of higher order thinking because it can be measured through validated and reliable instruments (Miri, David, & Uri, 2007).
Researchers have identified specific skills to define and assess CT competencies which include: evaluating evidence, logical reasoning, problem solving, discriminating, information seeking, predicting and transforming knowledge, all of which are important for college graduates to possess (Snyder & Snyder, 2008). These skills describe a person who is actively engaged in analyzing and evaluating information for meaning so that they may solve problems more effectively . Critical decision making (CDM) is one component of CT and measures students' ability to analyze information, form logical conclusions, and problem solve. This is essential for students because it establishes that they can consider facts, recognize gaps in evidence, and evaluate the choices when making decisions .
While CT and CDM are agreed to be important skills for students to develop there are barriers that prevent implementation of these learning practices. These barriers include time constraints in teaching course material, unfamiliarity with CT education literature, large class sizes, and low self-efficacy in utilizing evidence-based teaching practices (Gasiewski et al., 2012;Stieha, Shadle, & Paterson, 2017). Measuring outputs associated with CT, such as CDM, is also often over looked in college courses . There are many instruments designed to measure components of CT; however, they are time-consuming to administer and measure, costly, and vary in reliability and validity . Even with these barriers, it is essential to explicitly measure CT skills because this allows for understanding of the impact different teaching methods have on CT.
This paper focuses on the development of an intervention consisting of two online modules, using evidence-based instructional practices (constructivism, problem-based learning and scaffolding), designed to overcome barriers (large class sizes, time constraints, and measurement burdens) that prevent measurement and development of CDM skills. The results include an evaluation of the scoring system to measure CDM and outcome assessment of CDM.

Constructivism Approach to Learning
Constructivism, as a learning theory, is based on the concept that students learn by connecting concepts to previous knowledge and experiences . Three main principles of constructivism include: 1) knowledge comes from interactions within the environment, 2) a goal for learning encourages engagement, and 3) knowledge evolves based on social negotiations . Students are then able build knowledge based on what they already know rather than starting with a blank slate . The instructional principles of constructivism used in this study are summarized in Table 1. Provide support for reflection of content and learning process Allow the learner to analyze and make judgments related to the learning experience Overall motivation to engage and complete the task 1. Savery and Duffye, 1996 Problem-based learning Problem-based learning (PBL) provides students with opportunities to examine complex problems using a wide variety of resources, to develop their own strategies for addressing these problems, and to present and negotiate solutions in a collaborative manner . Teaching using PBL was first implemented in medical school curricula in the 1960s  and a meta-analysis shows that when compared to a standard teaching group, physicians who were exposed to PBL were found to be better prepared for their careers in terms of social and cognitive abilities . Nursing programs have also incorporated more PBL into their curriculum and have shown that it leads to stronger CT skills . The long-term relationship between PBL and CT are still being investigated, but there is evidence that PBL can increase specific skills related to CT such as problem-solving ability, investigating facts, and logical reasoning .
The ideal method for utilizing PBL is through dialectical reasoning, where the student is confronted with competing narratives . Social-problems are the most suitable for PBL activities because they are multi-logical and controversial . This can increase motivation to engage in learning because students are more invested in issues when they are centered around authentic situations, along with encouraging curiosity and attention (Yoo & Park, 2015). However, students have reported negative perceptions about PBL when the difficulty level of the problem seems unmanageable, and there is little guidance given by the instructor (Huang, 2005).

Scaffolding
The negative perception of "too difficult" can be reduced when scaffolding is provided to aid in problem-solving (Kim, Sharma, Land, & Furlong, 2013). Scaffolding is defined as support provided to guide students in completing complex tasks (Belland, 2014).
Scaffolding can be utilized to help students develop solutions and gather knowledge on their own rather than directing them towards specific information or a specific answer.
Scaffolding can be manipulated to provide more enhanced or decreased for minimal support.
This process of using enhanced and minimal scaffolding can be operationalized through technology where interactive activities are adjusted to provide more support or less support for the student when completing a task (Lajoie, Guerrera, Munsie, & Lavigne, 2001).

Overall design
This study was designed to measure and facilitate CDM in students who were in large introductory level classes. Students were randomized into either a CDM-framework (CDM-F) group or a control group. Both groups were exposed to CT videos that highlighted components and practices related to CT such as looking at information with an open mind and evaluating all the facts before making a decision. Both groups were also exposed to components of PBL using competing narratives where students watched videos on both sides of specific topic areas and were asked to make a decision. The following sections will cover the topic choices of the modules, the development of the CDM-F, and the development of the guided scoring system to measure CDM.

Topic Choice
The topics for the modules were 1) animal vs. plant-based sources of protein and 2) organic vs. non-organic food. The topics were chosen with PBL in mind and relate to authentic scenarios that college students deal with regularly. Both issues can be argued from multiple perspectives which implements the competing narratives approach. These competing narratives presented to students also progress in complexity, beginning with a more straightforward problem of animal vs. plant-based proteins, which has concrete evidence that plant-based proteins have less of an impact on the environment compared to animals, but differences in transportation and production methods are highlighted that confound the environmental benefits . Module two moves to a more complex issue of organic vs.
non-organic foods, where solutions are much more ambiguous ), compared to module one.

Information Collection-Competing Narratives
Information about both sides of the topic was presented in short videos using a competing narratives approach. The information is presented to the students and allows them to be the ultimate decision maker once they have gathered all the information. The competing narratives approach is a teaching method grounded in constructivism where evidence on both sides of an issue are presented, and the student can incorporate that evidence, along with previous knowledge and experience, into their decision-making process.

1) Topic introduced as a problem
The topics are presented as multi-logical problems, using competing narratives, that the students are responsible to solve. Multi-logical problems help aid in CDM by requiring students to examine alternative points of view and consider alternative solutions (Kim et al., 2013). The problems were framed using an authentic and relatable scenario to encourage engagement in learning, along with providing a goal for learning. The scenarios were also designed to challenge the learners' thinking and introduce differing viewpoints to prime the student to consider alternative views and perspectives.
Example of multi-logical problems: Module One: Your college promotes 'Meatless Mondays,' where they ask students to forgo meat once a week and choose a plant-based protein source. You are standing with two of your friends in the lunch line at the dining hall. One friend announces that they are going to eat the meatless option that night for dinner because it is better for their health and better for the environment. Your other friend criticized their food choice because meat is an excellent source of protein and avoiding meat once a week won't help the environment or their health. They start arguing about which option is best and need you to be the tie-breaker to settle the argument.
Module Two: Imagine that you have been living off-campus and share a house with two of your friends. For the past year, you have all been going grocery shopping together and sharing the cost of food between the three of you. It had been going well until one of your roommates decided to change their eating habits and will now only buy organic food because they heard that it is better for the environment and better for their health. Your other roommate argued that buying organic is just a waste of money and it is a marketing scam to make consumers spend more money on food. They debated about which option is best for the past week and now need you to be the tie-breaker to settle the argument.

2) Activity to organize information -Note taking activity
While viewing the videos, the students completed a note-taking activity. The notetaking activity was in the form of a t-chart that allowed the student to organize the information that they heard in the video into different sides that they felt it related to (e.g., organic vs. non-organic). The organization of data is what separates expert thinkers from novice thinkers (Bransford, Brown, & Cocking, 1999). The built-in support for organizing data allows students to use evidence more effectively in making decisions and to consider alternative viewpoints (Belland, 2014).

3) Scaffolding to help form a response -Mind Map
Scaffolds were implemented to help frame the argument before making a final decision. Asking students to process content while simultaneously expecting students to articulate their knowledge is difficult. By framing the argument, there is enhanced support for the organization of the decision while leaving the student to synthesize and reflect on their ability to think critically. The example of the enhanced scaffolding is in Figure 1. Both modules, one and two, included the mind mapping scaffolding in Figure 1, which helps the student organize the information before making their final decision.

Decision-Making Activity
The decision-making activity differed from module one to module two. Module one included enhanced scaffolding (Figure 2) of responses to aid students in developing a response that included three components: 1) decision, 2) evidence to support the decision, and 3) recognition of other sides perspective. These three components summarize aspects of CDM in which students are evaluating the evidence, identifying alternatives, and making a decision using evidence to support a conclusion. There is a reduction in scaffolding for the decision-making activity in module two (Figure 3). The students receive a prompt to make a decision and a blank text box. The goal of the reduction of scaffolding is for the student to implement what they have learned in module one, as far as forming a well-thought-out response using CDM skills and carry forth those skills to module two.

Rubric
The outcome analysis to measure CDM was designed to be efficiently and reliably scored ( Table 2). The scoring was developed by using a validated rubric that was originally designed to assess the use of CT skills when making a decision to buy organic vs. nonorganic milk . This rubric was transformed to an online guided scoring system to have the reviewer identify the different components of the response (decision, evidence-based reasoning, and alternative points of view). The student's ability to use CDM was evaluated on a scale of 0-30, with 0 representing a non-response/fails to address the task and 30 equating to a response that addresses the three following constructs: made a decision, used evidence to support the decision, and ability to see the other side's point of view. Student provides a recommendation and explains the decision, using supporting evidence. The recommendation refers to at least three points from the following dimensions: health value, animal care, cost, environmental impacts. The student discusses alternative points of view on the topic. 20 Student provides a recommendation and explains the decision but may use limited supporting evidence. The recommendation refers to at least three of the following dimensions of the topic: health value, animal care, cost, environmental impacts, but doesn't discuss alternative points of view on the topic. OR the student discusses alternative points of view but refers to less than three of the dimensions. 10 Student provides a recommendation but does not explain the decision, OR student explains solution but does not provide a recommendation. The recommendation refers to one of the following dimensions of the topic: health value, animal cost, environmental impacts. 0 Student does not respond or fails to address the task.

Interrater Reliability
Undergraduate research assistants (URAs) were trained to score the CDM responses, and their scores were compared to the scores of a senior researcher to calculate interrater reliability (IRR). After completing training on the different components of CDM, the URAs scored ten mock responses and had to receive an IRR score of >.80 to pass the training.
After passing the training, all responses were duplicate scored to assess reliability of the guided scoring system.

Sample
The modules were tested in large introductory science courses (introduction to human nutrition and introduction to agriculture sciences). Both classes had more than 150 students enrolled. Students were offered extra credit for participating in both modules.
Demographic data was collected to capture gender, age, and major.

Statistical Analysis
Interrater reliability was calculated using percent agreement. Duplicate scoring results were also assessed for frequency of agreement between scores. Demographic variables for participants participating in the modules were calculated using frequencies and means. Paired T-tests were performed to evaluate differences in aspects of the CDM scores for continuous variables and chi-square for categorical variables between the enhanced scaffolding condition and the minimal scaffolded condition.
A comparative sample with no framework was also assessed for within-group changes as a reference for scores that would be expected if the modules were designed without the CDM-F (topic introduced as a problem) or scaffolding (note taking, mind map, decision making support). However, the comparative group was exposed to aspects of PBL through the CT video and the topic videos examining both sides of the argument.

Inter-rater reliability and duplicate scoring
Following the methods of Horacek et al. (2013), URAs (n=4) participated in training until an acceptable 80% scoring agreement was met when compared to a senior researcher.
After passing training, they then scored a total of 440 responses independently, which were then duplicate scored. The duplicate scores where compared to assess for the reliability of the online guided scoring system (Table 3). The average time to score module one was 7 min/response. Module one matched response rate was 65% and when examining the unmatched responses, the majority were off by 5 points or less (72.3%).
The average time to score module two was 4 min/response. Module two matched response rate was 82% and when examining the scores that did not match, 82% were within a 5-point range. The outcome assessment of CDM was based on the module two score and because of this the two-phase scoring was found to be a reliable (>80% agreement). These results show that as the URAs gained more experience with scoring the responses, they increased their percent match rate and also decreased the time it took to score the responses.  Table 4 to illustrate a low, medium, and high score of CDM. Examining the responses, the low score response did not engage in the decision-making process, while the medium score response provided a decision and a reason to support their decision but did not recognize the other point of view. Finally, the high scoring response made a decision, used evidence to support the decision and recognized an alternative perspective.

Response Example
Neither one seems to be better than the other so choose which you think is best.
I would say to go organic because it is better for your health and helps with biodiversity.
I would tell my friends that the best option is to eat conventional. This is because it is a less expensive diet and has higher yields. It doesn't maintain biodiversity, but it uses less land. I can see why to go organic, but the cons outweigh its pros.

Impact of Enhanced and Minimal Scaffolding Compared to a Non-Scaffolded Condition
A total of 210 students completed modules with the CDM-F and 230 students completed the modules without the CDM-F. The majority of students were female (73.2%) and about 19 (±1.3) years old. Results for the CDM-F condition are reported in Table 5, and the reference group results are reported in Table 6.  Results show that in the CDM-F condition, from module one to module two, students significantly decreased their overall score and the number of evidence-based reasons that they referenced in their response. These findings are to be expected because module one was providing explicit instruction about what components should be in the decision (choice, three reasons, and the alternative perspective), whereas the module two condition removed that structure. However, there were no significant differences between module one and module two for making a decision or seeing the other sides point of view on the topic showing that the reduced scaffolding did not impact these aspects of the decision-making process.
In the non CDM-F condition students significantly increased their scores from module one to module two and referenced significantly more evidence-based reasons in their decision-making response but the proportion of students who made a decision decreased.
While the non CDM-F condition did not receive any scaffolding, they did receive instruction on CT and participated in evaluating evidence to make a decision by being exposed to the competing narratives. This exposure may have been enough to increase their CDM score.

Variable Mean (STD) (n=230) T-Test
However, in the CDM-F condition, even with a decrease in score from module one to module two, the final module two score was significantly higher compared to the non-CDM framework condition (results reported elsewhere). This brings into question as to what representation of the scaffolding is needed to effect students' abilities to use CDM skills.

Limitations
While the online modules were developed using theory-driven education methods to facilitate CDM, the length of the exposure to the content was limited. Furthermore, the scoring of CDM was designed with practicality in mind, but future research will need to be conducted to evaluate the reliability and validity of the scoring procedure using an online system. There was also a decline in the CDM-F groups ability to see the other side's point of view from module one to module two. While this was not found to be statistically significant, it is still important to consider when moving forward with PBL modules and helping students recognize other perspectives from their own.

Implications
Researchers and the workforce state that it is imperative that curricula be designed with higher order thinking skills in mind, but professors acknowledge barriers when implementing these methods, especially in large lecture settings (Stieha et al., 2017).
Utilizing technology to measure and facilitate CDM provides a feasible method for encouraging and capturing change in this skill set. The development of the two-part online module designed using a CDM-F was found to be easily administered and realistically scored for CDM by URAs.
The current development study demonstrates that it is possible to overcome barriers associated with CDM activities in large introductory classrooms. However, one of the greatest obstacles reported in the literature is related to measuring changes in these skills.  found that while faculty (n=18) reported using specific coursework for teaching CT skills, they did not explicitly measure those skills using validated or established tools. This is problematic because, as the authors conclude, "there is no way to know the efficacy of their efforts to develop CT in students" .
Alternatively, standardized test may miss gains in specific areas of CT abilities.  and , both used standardized CT assessments at the beginning and end of their classes. The courses were designed using PBL but found no significant increases in CT skills from baseline-post. However, both studies noted that based on assignments students collectively began to evaluate how they worked through problems and developed stronger metacognition skills, which may not have been captured within the constructs of the validated instrument.
The majority of existing CT instruments are time-consuming to administer and score. , integrated a similar approach as the current study by using a competing narratives approach in a large (n=209) introductory natural resource conservation course.
Pre-post CT ability was measured using a validated open-ended instrument. However, due to the time burden of scoring the responses, only 37 pre-post matched pairs were scored.
Students showed a significant increase in their CT skills and had scores significantly higher than national means but the time burden of scoring the responses demonstrated the challenge of measuring CT in a large class setting.
The current study overcame these barriers of measuring constructs related to CT by using a rubric that was integrated into a guided scoring survey. This was shown to have high reliability for the final outcome assessment in module two. The URAs showed improvement in their scoring, based on percent agreement, from module one to module two and decreased their scoring time. Future work focusing on the guided scoring system may want to implement more upfront training as evident by the improvement of percent matched scores from module one to module two. Overall, the guided scoring system overcame the barrier of capturing skills related to CT and was sensitive enough to measure changes between modules.
Results from the CDM-F group also showed that an enhanced scaffolding condition (module one results) was successful at guiding students to use CDM skills, and the majority of those skills were carried over into the minimal scaffolding condition (module two results).
These outcomes support those found by Kim et al. (2012) who used a pre-post design to measure the impact of two active learning modules, using scaffolding and PBL, on CT skills in a large introductory class. Critical thinking was measured using a rubric developed for the study as a composite score (1-6) and as a way to categorize thinking (emerging, developing, or mastering). Their results showed that students significantly improved their scores, but there was not a large enough increase to move them into a higher category of thinking.
These results are similar to the ones found in the current study, and demonstrates that scaffolding can work to guide CT or CDM, but students may need more exposure to scaffolding over longer periods of time to master decision making skills.

Future Research
It is important to stress that introductory courses are priming students for their future success as a student and future success at reaching their career aspirations, which is why designing learning activities to increase CT skills in these courses is important. Future research should continue to investigate how to incorporate evidence-based instructional practices, such as PBL, into large introductory courses and further develop ways to measure the impacts on learning outcomes (i.e., CDM). The modules described in this development paper should be further tested to explore the impact they have on encouraging motivation from students to participate in learning and how scaffolding can be expanded to prevent a decrease in CDM score. The scoring system for the modules should be replicated and evaluated in a separate sample to further understand the feasibility of measuring CDM in large classes.
Furthermore, skills that make up a strong critical thinker should continue to be evaluated to determine how the development of these proficiencies prepare students to be successful in the 21 st century. Exploring the relationship between CT skills and how it leads to greater preparedness and engagement of workers and citizens is important to investigate.
There are more and more young adults attending college every year, what are they gaining from their education and how prepared do they feel entering the job market? These are questions that are important to address within education research to facilitate change in how students are taught. Higher institutions are now, more than ever, focusing on strategies to increase undergraduate students' ability to think critically (Ahern, O'Connor, McRuairc, McNamara, & O'Donnell, 2012). Critical thinking (CT) is a component of higher order thinking, which involves analyzing, synthesizing, and/or evaluating information to solve problems (Scriven & Paul, 1987). Research has shown that students are not utilizing CT skills and are moving away from seeking out knowledge (Weiler, 2005). Instead, they are more likely to passively absorb information, have less curiosity in the unknown, and are hesitant to consider viewpoints that differ from their own (Weiler, 2005). While the majority of students enter college with weak CT skills, faculty also report little support from their departments regarding how to develop curricula that foster CT skills . This is concerning because teaching in terms of developing CT skills is an objective of many programs and influencing the type of thinkers that are moving into the workplace

Registration, Council on Future Practice, Education Committee, Nutrition & Dietetics
Educators and Preceptors DPG 2013; .
To be successful professionals, young adults need to apply and use CT skills when solving real-world problems as well as personal decision making, but employers are consistently reporting that recent college undergraduates lack strong CT skills . The 2013 "National Survey of Business and Nonprofit Leaders" reported that businesses emphasize a candidate's ability to critically think and solve problems more than their college major or grade point average (Desai, Berger, & Higgs, 2016). This highlights the need of colleges and universities to incorporate more training of CT skills and one way to do this is by incorporating more CT into courses. The purpose of this study was to develop an intervention that could be easily implemented into courses that would encourage students to use specific CT skills, ultimately preparing them for the 21 st century workplace (Rhodes, Miller, & Edgar, 2012;Wright, 1992).
While the importance of CT has been recognized in higher education , there is wide variability in its definition. Critical thinking skills are characterized by having the ability to apply standards, seek out information, problem solve, transform knowledge, predict consequences of decisions, be creative, practice logical reasoning, and evaluate evidence when faced with a problem or question . Critical thinking can also be measured through critical thinking disposition (CTD), or a person's natural inclination to use critical thinking skills . However, just because a person has a disposition to be a critical thinking does not mean that they acquire the skills necessary to implement that character trait (Stupnisky, Renaud, Daniels, Haynes, & Perry, 2008). The current study focuses on one component of critical thinking, critical decision making (CDM), which is characterized by having skills in problem solving, logical reasoning, and evaluating evidence when making decisions .
Unfortunately, limited research exists on how higher education can incorporate CDM skills into curricula. It is suggested that to facilitate CDM, instructors will have to move away from instructor-centered teaching methods, and towards learner-centered curricula (Brown, 2003). A learner-centered curriculum fosters CDM because it allows students to "construct knowledge by gathering and synthesizing information through inquiry, communication, critical thinking, and problem-solving" (Huba & Freed, 2000). One of the most successful ways of facilitating CDM is using problem-based learning (PBL). Problembased learning provides students with opportunities to assess complex problems using a variety of resources, develop their own strategies for addressing these problems, and present and negotiate solutions in a collaborative manner .
One method of PBL is to present students with opposing points of view . Social problems are ideal topics for PBL activities because they are controversial, where multiple perspectives can be taken into account .
This teaching method has been used in medical and nursing curricula , but little research has focused on science, technology, engineering, and math (STEM) courses.
Problem-based learning lends itself nicely to STEM topics because it allows learners to debate solutions to issues and evaluate multiple perspectives (Wals & Jickling, 2002).
One example of an issue facing society is the task of developing a more sustainable food system. This is an issue that can be debated in multiple ways from varying points of view but is an issue that needs to be addressed within the STEM field. Problem-based learning can help prepare students to solve these complex issues and be better prepared for future success in their careers by developing stronger problem-solving and decision-making skills (Dochy, Segers, Van den Bossche, & Gijbels, 2003).
Historically, curricula that have been developed to increase CDM in STEM majors has been integrated into semester-long classes (Harman et al., 2015;Heinrich, Habron, Johnson, & Goralnik, 2015;Lohse, Nitzke, & Ney, 2003;. However, developing semester long programs may not be feasible for professors who have a planned and tested curriculum that they feel works well for their students. Furthermore, it has been suggested that teachers aspire to teach in terms of higher order thinking but may lack the necessary techniques, methods, and resources to foster that type of learning . Barriers related to the feasibility of incorporating strategies to enhance CDM skills underscores the need to develop a framework to facilitate CDM activities into STEM courses that can be easily integrated into existing curricula. Thus, the objectives of this study were to understand if the addition of a contextual CDM framework (CDM-F), to a two-part online nutrition module focusing on sustainable eating, influences college students' CDM and to explore the relationship between sustainable eating behavior ("green eating"), CTD, and CDM score. Problem-based learning modules framed around sustainable eating behavior is an appropriate context to encourage CDM because of the inherent complexity of the problems related to matters such as plantbased vs. animal-based protein, and the abundance of conflicting messaging related to organic vs. non-organic foods (Lim et al., 2017). The primary hypothesis was that undergraduate students exposed to the CDM-F would have a significantly higher CDM score when compared to the students in the control group. The exploratory hypothesis was that green eating behavior and CTD would be mediators of CDM score.

Critical Decision-Making Framework
The CDM-F was designed using the critical constructivism model . The development of this model aimed to provide learning tools to bring about a conceptual change in student thinking by allowing the student to construct their own conclusions when presented with information . The three specific strategies included to operationalize an online delivery of this method of teaching, are as follows: 1) Topic is incorporated and introduced as an authentic problem related to foods that students consume each day (Chaillé, 2008), 2) information collected from the two sources with alternative views is organized using an input scaffold, or support, in the form of a t-chart which helps facilitate the organization of information, and allows students to use evidence more effectively in making decisions, and to consider alternative viewpoints (Brand-Gruwel, Wopereis, & Vermetten, 2005), and 3) an output scaffold is provided to help frame their arguments separate from decision making in the form of a "mind-map". The mind-map was used to help the participant form their case before their final decision (Wheeler, 2003). The CDM-F provided more structured support in the first module, and general support in the second module to move students towards independence in CDM.

Topic Choice
The topic choice for the modules included 1) animal vs. plant-based sources of protein and 2) organic vs. non-organic food. The topics were chosen with PBL in mind and relate to authentic scenarios that college students are dealing with every day. Both of the issues can be argued from multiple perspectives.

Research Design
This study was a randomized control-trial, where students from two universities were randomly assigned to the CDM-F group or the control group. One university was located in Rhode Island, and the other was located in Texas. The study was approved at both universities through the instructional review board. Students were provided with a link to access and sign-up for the program and randomized into either the CDM-F group or control group by the computer. Following the consent process, students were directed to an online pretest, which assessed CTD, green eating, and basic demographic data. They were then immediately directed to the first module, which covered the pros and cons of animal protein vs. non-animal protein foods, and then one week later completed the second module, which included the pros and cons of organic foods vs. non-organic grown foods. The differences in module one and module two between groups are described in Table 1.

Module Overview
Modules were easily accessible to students from various platforms via internet connection, including mobile devices and desktop computers, and took about 15 minutes to complete. Module one began with a fun and interactive "quiz" to determine what type of learner the student was, a video discussing CT, and two videos addressing both sides of the specific topic area (animal protein vs. non-animal protein foods). Next, after watching the topic video, the student was asked to make a decision about which side they agreed with and then prompted to write a brief response explaining why they made that decision. One week later, the participant was notified through an announcement that module two was available.
Participants completed the same log-in procedure as in module one and were directed to module two (organic foods vs. non-organic grown foods). Module two had the same format as module one and then was followed by the posttest.
For the CDM-F group, the scaffolding for the decision-making activity (strategy three of operationalizing an online delivery method to teach CDM), was reduced from module one to module two. The scaffolding for the final decision-making activity in module one was designed to be more intensive by providing fill in the blanks that prompted the student to use evidence-based reasoning to support their decision and recognize the other side's point of view, compared to the reduction in module two, which replaced the fill in the blank prompt with an empty text box for the response. To analyze the influence of the CDM-F vs. control condition between groups on CDM, the response from module two was scored using a rubric.

Measurements
Critical Decision-Making Score To calculate the CDM score, a previously developed and tested rubric  was modified to score the decision-making activity at the end of each module.
Scores ranged from 0-30, evaluating the extent of CDM, with 0 representing a nonresponse/failed to provide a text response addressing the task, and 30 indicating a text response that addressed the three following constructs: 1) ability to make a decision (0/10 points), 2) evidence to support the decision (0-15 points), and 3) ability to see the other side's point of view (0/5 points). The responses were scored using a computer-assisted scoring system that guided trained researchers through the response criteria generating a total score. The passing rate for training was set at an IRR of >.80 (Horacek et al., 2013).
All responses were duplicate scored by research assistants (n=4) who participated in training and were evaluated for reliability based on matching rate. Scores that did not match were then scored by a senior researcher to determine a final score.

Critical Thinking Disposition
Critical thinking disposition was measured using a 5-item CT subscale from the Motivated Strategies for Learning Questionnaire (MSLQ) (α=.72) (Pintrich, Smith, García, & McKeachie, 1993). The CTD subscale was designed and validated to measure college students' thinking strategies to apply knowledge and critically evaluate situations (Stupnisky et al., 2008). The composite score is an indicator of students' natural inclination to use CT during a decision-making process (Pintrich et al., 1993). The five items were measured using a 5-point Likert scale, ranging from (1) strongly disagree (5) strongly agree, for example, "I often find myself questioning things I hear or read in my courses to decide if I find them convincing". Critical thinking disposition score was assessed at baseline and after module two.

Green Eating Behavior
Green Eating (GE) behavior was measured using a 7-item survey (α = .81), which assessed the frequency of choosing sustainably produced food . The items included behaviors related to purchasing foods locally grown, shopping at farmer's markets, buying organic, and purchasing free-range animal proteins. Items were measured on a 5-point Likert scale ranging from (1) barely ever to never to (5) almost always, a higher score indicating a greater frequency of choosing sustainably produced foods.

Data Analysis
Data were analyzed using SPSS 24 and EQS (structural equation modeling software).
Descriptive variables were analyzed for normal distribution using skewness and kurtosis.
Independent t-tests were used to analyze baseline differences between the CDM-F and the control group, and between universities. A Chi-square analysis was used to analyze categorical variables. To assess differences in CDM score between groups, a two-stage hierarchical multiple regression was performed with CDM score as the dependent variable.
University was entered at stage one of the regression to control for differences at baseline.
The group (control vs CDM-F) was entered at stage two to assess differences between groups on CDM score over and above university. To evaluate change in CTD score and GE behavior from baseline to post-intervention, a repeated measures analysis of covariance (ANCOVA) was conducted. An exploratory structural equation model using path analysis was also performed to explore the amount of variation that was accounted for in CDM score by group, CTD, and GE behavior.

Sample
A total of 440 students randomly assigned to either the control (n=230)  Significant baseline differences between universities were found for all categorical and descriptive variables (Table 2). Based on these findings, university was controlled for in the comparison of group differences and CDM score to account for differences. Baseline comparisons between groups are shown in Table 3. At baseline, the control group initially had a significantly higher pre-CTD score (p<.05), but after controlling for university, this was no longer significant. No other differences between the CDM-F group and control group were found.

CDM Score
The hierarchical multiple regression revealed that at step 1, university contributed significantly to the regression model, F (2,429) = 11.97, p< .001 and accounted for 5.0% of the variation in CDM-score. Introducing the grouping variable (CDM-F group vs. control group) explained an additional 3.0% of the variation in CDM score and the change in R 2 was significant, F (2,432) = 18.74, p<.001. The results show that after controlling for university, the grouping variable explained a significant additional percent (3%, p<.001) of the variation in CDM scores, demonstrating that the CDM-F group had a significantly higher CDM score than the control group. The regression statistics are reported in Table 4. Group differences presented in Table 5 show a significant difference between the scores were explained by a greater percentage of the intervention group making a decision and also used significantly more evidence-based reasoning to support their decision when compared to the control. However, there were no significant differences between groups in recognizing the other side's point of view. After controlling for university, no differences in CTD or green eating over time between groups were observed (Table 6). However, there was a significant within-group change for both the intervention and control, where both groups increased their CTD score from baseline to post-intervention. No differences in GE behavior between or within groups were observed (Table 6). For the exploratory structural equation modeling path analysis, three model versions were hypothesized and tested using EQS: direct, predictive and mediational model. Findings revealed that compared to a direct model with only a single predictor from group to CDM score, and a mediational model with CTD and GE behavior as mediators, a third prediction model with paths from three predictors (group, CTD, and GE behavior) fit best.
Macro-level fit indices showed that the c 2 , df, CFI, and RMSEA were all in a nearoptimal range for the selected prediction model (Table 7). In contrast, fit indices for the direct effect and mediational pathway indicated that these models were not adequately describing the data. Standardized maximum likelihood parameter estimates for the prediction model path coefficients are shown in Figure 1, along with R 2 values. The results indicate that group and GE behavior were significant predictors of CT score, but CTD score did not significantly explain the score. that of other interventions that were implemented in semester-long courses (Şendağ & Ferhan Odabaşı, 2009), highlighting that these types of online modules can be used as supplemental instruction to support higher order thinking skills in accordance with knowledge learned in the classroom.
The successfulness of the online interactive modules in facilitating CDM can be attributed to grounding the design within the constructivism theory of PBL and scaffolding the learning to help students move towards independence in their CDM skills. Social constructivism and PBL require the student to play an active role in the learning process, while, the use of scaffolding can help students overcome perceptions that a topic is too difficult to understand. Scaffolding aids in learning by providing examples and structure about how to solve a problem. The findings in this study are reinforced by , who implemented similar methods using PBL activities throughout a semester-long course in a large introductory class. Students were exposed to competing viewpoints of topics related to sustainability and were assessed on CT skills using a standardized open response test (The Critical Thinking Assessment Test). For , CT skills significantly increased in students from the beginning of the class compared to the end, but only a small portion of the matched pair responses (18%) was able to be scored due to the research burden of evaluating the scores. Similar curricula design should be intentionally integrated into higher education STEM courses to support CDM skills in students, but there needs to be a feasible and explicit way to measure CDM.
While improving CT is often an objective of a college course, it is usually not explicitly measured as an outcome (Harman et al., 2015;. One reason for this is because it is difficult to define and measure .
Validated instruments, which measure CT, are often lengthy and time consuming for the participant which prevents practical use in the classroom . The current study offers a method of measuring a component of CT, which is CDM, that can easily be used in large introductory level classes and was found to have strong interrater reliability. Undergraduate research assistants were able to accurately score the CDM responses and had a high agreement when assessed by duplicate scoring. This shows the feasibility of explicitly measuring CDM, even in large introductory classes.
There were no differences between groups in addressing the other side's point of view. This is consistent with the current literature that college students are rigid in their opinions; especially around controversial topics that challenge their existing viewpoints (Weiler, 2005). For example, Trosset (1998) conducted a qualitative study with 200 undergraduates and asked if topics related to diversity issues could be discussed with both sides being represented equally. Students were not interested in discussing the topics unless they had a strong opinion about a side, and their motivation to engage in the discussion was to influence their peers to think similarly to themselves and were not motivated to listen to their peer's point of view if it differed from their own. However, other studies have found that discussions, even when implemented online, can help students see other perspectives than their own and critically evaluate evidence . Based on the current study, an extension might be useful with an integration of a discussion activity, which may increase the rate at which students recognize a differing perceptive than their own.
The current study presented no significant changes between groups in CTD, but the mean scores were found to be similar to other studies focusing on college students (Pintrich, 1991;Shastri, Wang, & Gandhi, 2015). There were no differences between age, gender, or major in relation to CTD score, which supports previous research that these demographic variables are likely not related to CTD (Domenech & Watkins, 2015;Hunter, Pitt, Croce, & Roche, 2014). However, CTD did significantly increase in the intervention and the control over time. This can be explained by the control group being exposed to components of PBL instruction such as the CT video, the competing narratives in module one and module two, and the decision-making process. Alternatively, researchers using similar methods found that problem-solving skills and CTD had a small relationship, and presented the concept that problem solving might be a better indicator of decision making overall, rather than related to having a higher or lower CTD (Friede, Irani, Rhoades, Fuhrman, & Gallo, 2008). The relationship between CTD and decision making and, how these attributes explain CT, should be further explored within this population.
The exploratory findings from the path analysis show that the CDM-F and GE behavior were significant predictors of CDM score, while CTD was not. Other research findings have shown that when students relate more to a topic, they have a greater motivation to engage in learning activities and CT in low stakes assessment (e.g., extra credit) (Liu, Bridgeman, & Adler, 2012;Weiler, 2005). This may explain why GE behavior was a significant predictor of CDM score because the topics chosen were related to sustainable eating practices. The students on both ends of the GE spectrum may have a stronger desire to express their point of view in the decision-making activity, therefore scoring higher by making a decision and referencing evidence to support their decision.
While this may have led to a higher CDM score for those reporting higher GE behavior, it may have hindered CDM scores for those students who were not as interested in the subject matter. Therefore, future research wanting to implement similar methods as this study may want to consider students level of interest or current behavior related to the context of the PBL activity, as it could be a motivating or discouraging factor to engage in a deeper level of learning, especially in a low-stakes learning activity.
The path analysis also revealed that CTD score did not account for a significant amount of variation in CDM score. This is conflicting with other studies that have found that CTD is an indicator of CT skills (Pascarella, Terenzini, & Feldman, 2005), however it is important to note that the main outcome for this study measured decision making skills, not the larger construct of CT, which is a possible explanation as to why CTD was not a significant predictor of the outcome. Furthermore, having a natural inclination to be a critical thinker does not equate to having strong critical thinking skills (Stupnisky et al., 2008). For example, students may have recognized the need to use CT in their courses (i.e., measured by CTD score) but may not have progressed to the stage of implementing those strategies (i.e., measured by CDM score). Having knowledge about a behavior without implementing that behavior has been found throughout the literature when looking at educational theories (Chaill, 2008) and theories about behavior change, such as the Stages of Change Model (SOC) (Prochaska, Redding, & Evers, 2015). For example, the SOC theory states that people need to be exposed to a situation multiple times and progress through stages that are designed to help move them towards independence in order to turn beliefs into measurable behaviors or outcomes (Greene & Rossi, 1998). Therefore, while students in the intervention group achieved a higher CDM score compared to the control, the intervention may not have been intense enough in dose or exposure to cause a change in their CTD when compared to the control (Giancarlo & Facione, 2001).
The previous discussion brings forth limitations. This study did not measure the broader construct of CT, which limits the ability to completely understand how the CDM-F impacted students' ability to think critically. Measuring CT would have led to a deeper understanding of the influence of the CDM-F on CT skills, but there are disadvantages of administering a CT assessment such as, an increase in participant burden, the time it takes to score the responses, high cost associated with many of the instruments, and the questionable reliability/validity of the instruments (Abrami et al., 2015;. Additionally, the current study did not include a discussion activity among students, which may have been needed to encourage the recognition of other perspectives and to identify the other side's point of view. Previous research has shown that discussion activities are successful at helping students see other perspectives that differ from their own .
Despite limitations, this intervention used a rigorous study design and introduced a novel approach that facilitated CDM skills by using an online interactive framework that can be easily administered online or in a classroom setting and be manipulated to fit multiple topics within the STEM fields and beyond.

CONCLUSION
Having the ability to use CDM skills when organizing information and forming conclusions using evidence-based reasoning is imperative for college students to master as they begin their careers. As future professionals, this generation faces challenging problems in the coming decades that are unpredictable. Outside of the workplace, one major issue facing the next generation is environmental and food system sustainability, which will require CDM skills and innovation to develop solutions to this multifaceted problem. While the young adult generation recognizes that information is more abundant than ever, the use of the internet and the popularity of technology requires information seekers to be critical of the facts they consider when making complex decisions. Future research should aim to further develop mechanisms that encourage young adults to explore alternative viewpoints and acknowledge that there may be multiple solutions to an issue. However, as evident by this study, using a framework specifically designed to engage the learner and provide support to form a decision was found to be a facilitator of CDM, and students demonstrated greater implementation of these CDM abilities when exposed to the CDM-F within the two online interactive modules.

Role of Higher Education Nutrition
The Academy of Nutrition and Dietetics (The Academy) reports that it is the profession's responsibility to anticipate changes in the health system and to prepare students for future career paths as professionals who will guide nutrition and wellness for the

Experiential learning
Experiential learning theory is learning through experience and reflection where students apply what they already know and their understanding of facts to real-world problems (Kolb & Kolb, 2012;Mughal & Zafar, 2011). Experiential learning takes into account that students often draw on their own daily experiences when solving problems, for example, interactions that they have had with friends, family, and situations embedded in their environment (Mughal & Zafar, 2011). This type of learning can be especially beneficial in nutrition education because if students are exposed to authentic problems that may arise in their career, then through experiential learning, they theoretically will be better prepared in problem-solving, communication, and overall ability to critically analyze the situation. In the future, nutrition professionals will be faced with challenging problems that are unpredictable. One major issue facing the next generation of dietetic professionals is going to be focused on helping consumers navigate a sustainable food system.

Sustainable Food System
For the purposes of this review "sustainable food system" is being defined as, "the ability to meet current needs of food production without compromising the ability of future generations to meet their needs and is critical to every aspect of the food system including production, processing, distribution, consumption and disposal of food" (Monroe, Lofgren, Sartini, & Greene, 2015). Consumers report that they are interested and drawn to products that are "environmentally friendly" (Pelletier, Laska, Neumark-Sztainer, & Story, 2013), and positive attitudes toward organic, local, and sustainable foods are associated with higher dietary quality among young adults (Pelletier et al., 2013). However, aspects of the food system such as, excessive use of natural resources, large amounts of land use, water use, and pollution create a system that is not preforming in a sustainable way (Garnett, 2014;Monroe et al., 2015). When looking for solutions to increase sustainability it is important to think about the food system from both a production and a consumption standpoint and that currently the balance between these two facets is unequal. On one hand, it is necessary to produce enough food to feed the growing population, but on the other hand, it is necessary for those people to have access to foods that will promote health (Garnett, 2014).
Food production and access to food is an issue that is facing people globally, highlighted by the fact that there is currently the highest proportion of malnourishment recorded (Pimentel & Pimentel, 2003). The growing reality is that the food system is not sustainable because of the dependence on fossil fuel energy, fresh water, and the depletion of land area (Pimentel & Pimentel, 2003). Examining the food system as a whole, it is clear that there remains a complex array of issues, such as importing feed for livestock, down to the way a potato is prepared for a meal, that contribute to the global footprint the food system has on the environment (Davis, Sonesson, Baumgartner, & Nemecek, 2010).
The food system could be more sustainable by rethinking production efficiency through increasing yield capacity, reducing food loss, expanding aquaculture, and also by a demand restraint outlook of modifying diets (Garnett, 2014). These changes highlight the importance of technological and managerial improvements which have the potential to reduce environmental strain by decreasing carbon emissions, resource use, and waste output while increasing supply (Capper, 2011;Garnett, 2014). However, consumers have negative perceptions associated with improved efficiency via technology. For example, while genetically modified organisms (GMOs) and pesticides are used in a large number of products, public perception of these practices are not widely accepted (Gipmans, Schoeneboom, Klein, Bihlmeyer, & Saling). These perceptions exist and have proven difficult to change (Lucht, 2015) even though GMOs have seen to lead to a decrease in greenhouse gas emissions, one example of this is by limiting the fuel used to apply pesticides (Brookes & Barfoot, 2014). Nevertheless, even with an increase in efficiency and technology, the food system will continue to drive to meet consumer demands. Currently, based on consumer preferences, there is an apparent demand for high amounts of meat and dairy production, which results in a rise in agriculture emissions (Friel et al., 2009).

Animal based protein compared to plant based
While consumers world-wide demand and value animal-based protein sources (Reijnders & Soret, 2003), there is evidence to suggest that plant-based sources have less of an impact on the environment. Altering diets to contain predominately plant-based protein shows to be more beneficial to the environment by requiring less natural resources (i.e., fossil fuel energy and land) when compared to a meat-based diet (Pimentel & Pimentel, 2003). Even when production methods are taken into account, beef production has the greatest impact on the environment when compared to vegan and vegetarian diets and is a high contributor to water pollution and deforestation (Baroni, Cenci, Tettamanti, & Berati, 2007). Production of beef also uses up the greatest percent of available fresh water on the planet, accounting for about 70% of the utilization, most of which is used to produce crops for feed (Baroni et al., 2007). However, while evaluating environmental impacts it is important to consider where the food was produced and how far it had to travel to reach the consumer because differences in transportation can confound environmental benefits associated with plant-based diets . Using a life-cycle assessment (LCA), Jungbluth et al. (2000) found that 1kg of plant-based food transported by air is comparable to the negative environmental impacts of producing 1 kg of organic beef. Similarly, the deep freezing of vegetables has also shown to have comparable environmental impacts to the production of beef (Jungbluth, Tietje, & Scholz, 2000).
There are also other meat-based protein options to consider other than beef such as, chicken and fish. When looking at green-house gasses (GHGs), plant-based diets were found to produce the lowest levels of GHG emissions, however chicken and egg production were found to be produced at lower levels when compared to beef (Carlsson-Kanyama & González, 2009). For example, Pimentel and Pimentel (2003), reported that broiler chickens and turkeys required less fossil fuel to protein output when compared to milk and egg production (Pimentel & Pimentel, 2003). The environmental impact of aquatic systems, specifically fish, is important to consider also. The majority of fish consumed are caught and while less resources are needed (e.g., land, fertilizer, fresh water), production does require the use of fossil fuels (Reijnders & Soret, 2003). It is estimated that based on a trawler fishing system, catching fish requires 14-65 times more fossil fuels compared to producing plant-based proteins (Reijnders & Soret, 2003). However, when examining non-trawling fishing systems there is much less GHG emissions (Clark & Tilman, 2017). These nuances within protein sources highlight the importance of consumers to consider production and management practices when aiming to make sustainable choices in their protein selection.

Organic food production compared to non-organic food production
Another food production issue this is often debated is organic food production compared to conventional food production. Even though organic land production accounts for only about 0.9% of all land used for growing food, there is an increased demand and a growing market driven by consumers . Organic production methods are defined by the USDA to be ones that foster the cycling of resources, promote ecological balance and conserve biodiversity, with the overarching goal of maintaining or enhancing soil and water quality, while conserving wetlands, woodlands, and wildlife. Consumers often report positive perceptions of organic production methods and foods in regard to an increase in health value and beneficial environmental impacts (Bourn & Prescott, 2002). However, there is conflicting evidence as to whether these perceptions are supported in the literature.
Regarding environmental impacts, organic farming has found to promote greater biodiversity and soil health when compared to non-organic production methods . Focusing on biodiversity, organic methods on average have found to promote between 30-50% higher species and flora/fauna abundance, resulting in increased soil health and a greater prevalence of pollinators ). This promotion of biodiversity has been related to the restriction of pesticides and herbicides that organic farmers must adhere to. A recent LCA by Clark and Tilman (2017), explored the environmental impacts of 46-paired organic-conventional production methods.
The findings showed that organic production required more land (25-110%), had higher eutrophication potential, or greater nutrient runoffs, and resulted in similar GHG emissions as conventional farming (Clark & Tilman, 2017). The authors relate the findings of higher eutrophication and GHG emissions to the use of manure as fertilizer in organic systems, compared to synthetic fertilizer which are used in conventional systems. The amount of nutrients in the soil deposited by manure may be more than the plants need or use at given time point, which results in excess nutrient run-off in water that then can lead to aquatic dead-zones or algae blooms. The excess nitrogen in the soils, deposited by manure, also turns into nitrous oxide which contributes to GHG emissions at similar levels as conventional food production. However, Clark and Tilman (2017) also found that by not using synthetic fertilizers and pesticides there was less energy used in organic production and that consumers health may benefit from consuming organic foods over conventional foods.
Consumer perceptions around organic foods often circulate around perceived health values of increased nutrition and decreased exposure to pesticides (Aschemann-Witzel, Maroscheck, & Hamm, 2013). Hunter et al. (2011), reviewed 66 studies and evaluated the levels of micronutrients and minerals in organically grown produce compared to conventionally grown produce. Their findings showed that organic produce had significantly higher levels of overall total micronutrients and minerals (5.7%) when compared to their conventionally grown counterpart. However, when micronutrients and minerals were looked at separately, only mineral content was significantly higher in organic foods when compare to conventional. A meta-analysis was also conducted by Smith-Spangler et al. (2012), which investigated the health safety of organic versus conventional foods. The authors reported on 223 articles that focused on nutrient levels in foods and found that there were no significate differences in micronutrient levels between organic and conventionally produced products (plant-based and animal-based), with the exception of phosphorous. Comparisons did show higher levels of omega-3 fatty acids in organic milk and chicken, however the authors noted that there were few studies (n=5) that examined fatty acids and that they were heterogeneous. Overall, organic food and conventional food were found to be very similar based on nutrient profiles.
Another health factor that is often referenced by consumers as a benefit to eating organic foods is decreased exposure to pesticides and chemicals that are used in conventional farming practices. Unfortunately, there are few long-term studies that have investigated the relationship between pesticide exposure and health outcomes (Smith-Spangler et al., 2012). Although, those that have explored possible effects of pesticides on immune and allergy function have shown no negative outcomes (Kummeling et al., 2008).
While, more research needs to be conducted to further investigate the health differences between organic and conventionally grown foods, consumer perceptions continually refer to improved health and environmental benefits of organic products versus ones that are conventionally grown.

Green Eating Behavior
One challenge in creating a sustainable food system originate from dietary behaviors of consumers, which highlights the importance of future dietetic professionals to have a critical understanding of food production methods and its contributions to environmental degradation . Individuals sustainable eating behaviors are determined by their knowledge, values, attitudes, and willingness to act. Internal factors such as, attitudes and values, along with external factors such as, social environment, infrastructure, and perceived sacrifice, all influence environmental behavior changes. Ultimately, consumers have a lot of influence over what and how food is produced and made available. This underscores the importance for scientist to effectively communicate evidence-based information to the public, so consumers can make informed purchasing decisions.
Overall, communicating information to consumers about sustainable food production can be challenging due to existing preconceptions and emotional ties related to this type of food such as, "more wholesome", "tastes better", and "healthier" (Pelletier et al., 2013).
However, while these may be misperceptions related to sustainable food production, positive attitudes toward organic, local, and sustainable foods are associated with higher dietary quality among young adults (Pelletier et al., 2013). To measure sustainable eating as a whole, , developed and validated a tool assessing environmentally conscious eating behavior, defined by "Green Eating" (GE), or limiting processed foods and consuming locally grown foods, seasonal produce, fair trade certified or certified organic foods and beverages, meatless meals weekly, and animal products free from hormones and antibiotics. Results concluded that self-efficacy for GE depended on an individual's school and home environments and GE behaviors varied by individuals' readiness to adopt a GE lifestyle . However, while perceptions around sustainable eating behavior has been summarized and measured, there are few studies looking at how to increase these behaviors. Monroe et al. (2015), developed and tested four online modules that focused on increasing GE behavior in undergraduate college students (n=607) using a quasiexperimental design. The modules focused on teaching about GE, encouraging the purchases of locally grown foods, reducing food waste, and choosing environmentally conscious proteins. The results found that the students in the experimental group significantly increased their behavior, knowledge, and positive attitudes about GE when compared to the non-treatment control group. These findings highlight the important role knowledge can have on behavior practices. If complex issues, like sustainable eating, are communicated in a way that is straight forward then it may lead to better adoption by consumers. However, this requires professionals to deliver evidence-based messaging accurately and effectively to relay recommendations to consumers.
Overall, increasing critical thinking skills in future dietetic professionals through experiential learning can help lead to better health care for consumers and innovative solutions in the future. Students need to be leaving college with strong critical thinking skills because they will be the future professionals responsible for developing innovative solutions to complex issues, such as the challenge of creating a more sustainable food system.

Agree Strongly Agree
Choose not to answer 4. I try to play around with ideas of my own related to what I'm learning in my courses. Strongly Disagree Disagree Neutral Agree Strongly Agree Choose not to answer 5. Whenever I read or hear an assentation or conclusion in a class, I think about possible alternatives. Strongly Disagree Disagree Neutral Agree Strongly Agree Choose not to answer Green Eating Stage of Change 1. Green Eating is: Eating locally grown foods, limited amounts of processed/ fast foods, eating meatless meals at least one day per week, choosing organic foods as much as possible, and only taking what you plan on eating." Are you a Green Eater? No, and I do not intend to start within the next 6 months No, but I am thinking about becoming a green eater within the next 6 months No, but I am planning on becoming a green eater within the next 30 days Yes, I am a green eater and have been for less than 6 months Yes, I am a green eater and have been doing so for 6 months or more Choose not to answer