Process of Evaluation of an EFNEP-Enhanced PSE Intervention in Urban Schools

Objectives: To determine to what extent the EMPOWER intervention was delivered as originally planned and how participants perceived its delivery. Methods: This was a process evaluation study; data was collected using fidelity and observation checklists, grading rubrics, focus groups, semi-structured interviews, and meeting minutes. Program fidelity was assessed by calculating percent average of curriculum delivery. Program perception was assessed using the subjective data recorded on the fidelity checklists and responses from focus groups and semistructured interviews. Qualitative data were analyzed to detect common themes using NVivo11 Software. Results: The intervention was well received by students, school staff, and foodservice. Implementation was high, 97% of the curriculum objectives were met on average. Sixty-four percent of the take-home assignments were turned in. Ninety-four percent of enrolled students participated throughout the intervention. The evaluation identified several areas for improvement, lessons should be shortened and simplified and communication with classroom teachers should be improved. Conclusion and Implications: The EMPOWER intervention was successfully implemented with a high degree of fidelity, dose, and reach and was positively perceived by all stakeholders. Additional comprehensive process evaluation studies are needed to identify areas of improvement for future implementation of effective PSE-change interventions.


INTRODUCTION
The growing rate of childhood obesity and its association with serious medical consequences have created the need for sustainable evidenced-based interventions to prevent childhood obesity, particularly among low-income and ethnically diverse populations who are at a higher risk. 1 Given the important role that the environment has on the development of obesity, public health interventions are increasingly implementing strategies involving policy, systems and environmental (PSE) change. 2 Policy, systems and environmental change interventions focus on multi-sectorial levels of influence to change and sustain healthy behaviors in communities by applying socio-ecological theories. 3 In contrast to individual or small group interventions, PSE change programs offer strategies with greater population impact than individual change strategies by making healthy choices the easiest and most convenient choice. [4][5][6] However, descriptions of their implementation and evidence of the effectiveness of PSE interventions is still lacking, particularly among school-aged children. 7 School settings are now considered to be a viable location for PSE interventions. 8 Previous reviews of school based interventions have demonstrated the effectiveness of a variety of different approaches to improve dietary behaviors, and some of these interventions aimed at modifying school policies and environments. 9,10 Although PSE interventions are now considered to be most effective for public health, more studies are needed to establish a strong evidence base for the process by which PSE change interventions are effective. [2][3][4] Outcomes research as well as process evaluation research of PSE interventions is needed to address this research gap. Process evaluation is used to monitor and document program delivery and can help explain program outcomes. 11 Recently, emphasis has been placed on the importance of process evaluation of PSE change programs; however, research has been based primarily on their outcomes rather than how programs accomplish their goals. 12,13 Outcome evaluations determine whether an intervention was successful or not. 13 Process evaluation is used to document and determine to what extent a program was implemented as designed and can aid in understanding why it was or was not effective. 11 Process evaluations help explain whether specific elements such as fidelity (how well the intervention was delivered as intended), dose (to whom it was delivered) and reach (the extent to which the target population was reached) could affect program impact and outcomes and can help fine-tune program components. 11 Process evaluations gather data on the social processes involved in the delivery and reception of the intervention and use survey questionnaires, structured or semi-structured interviews, attendance logs, checklists, inventories, focus groups and direct observation. [13][14][15] Reviewers have found that interventions often focus more on documenting outcomes and less on process evaluation, which are needed to better understand the barriers and facilitators of achieving PSE changes and provide comprehensive guidance to future studies. 2,3,7,15 Recently, more school-based interventions have begun to include process evaluation in their studies. [15][16][17][18][19][20][21][22][23][24][25][26][27][28] Given that some school-based interventions have only achieved moderate success in changing dietary behaviors, process evaluations measuring how well strategies were implemented can help provide direction for increasing program effectiveness in the future. 18 The purpose of this study is to conduct a process evaluation of a school-based PSE intervention on increasing fruit and vegetable intake in fifth-grade children from lowincome, ethnically diverse schools.

METHODOLOGY Overview
This project was a process evaluation using data collected from a one-year school-based intervention called "Empowering Urban School Children to Increase Fruit and Vegetable Consumption through EFNEP-Enhanced PSE Interventions" (EMPOWER). This study was designed to determine to what extent the program was delivered as originally planned and to explore perception by students, staff, and other stakeholders. This process evaluation study was planned following a comprehensive guide described by Saunders et al. 11 An overview of the methodology and instruments used can be found on Table 1.

Participants
The EMPOWER sample included fourth-grade classrooms at four urban schools in Pawtucket, Rhode Island which are serviced by Aramark foodservice. Two treatment schools and two control schools were selected by the research committee. American with 76% from low-income households. 29 Six students from each school (total of 12 students), two school principals, and three health teachers at the two experimental schools receiving the PSE intervention were also included as part of the process evaluation data. As well as one Aramark foodservice assistant manager, three Expanded Food and Nutrition Education Program (EFNEP) nutrition educators, and members of the Pawtucket Wellness Committee.

Procedure and Description of the Study
As part of the EMPOWER intervention, the following data were collected for the process evaluation study. Data were collected pre-, post-and during the intervention spanning from September 2015 to May 2016.
The process evaluation of the EMPOWER intervention consisted in determining to what extent the curriculum was delivered as planned. The program was made up of 10 lessons designed to be delivered every other week over a period of 20 weeks. Each lesson was developed to build upon an existing URI SNAP-Ed FFVP curriculum consisting of 8 lessons that focused on nutrition education to increase fruit and vegetable consumption in elementary school students and is designed to be taught by classroom teachers. The PSE lessons, delivered by trained EFNEP educators, were designed to be delivered in alternating weeks with the SNAP-Ed FFVP curriculum.
The PSE lessons were planned to be taught during 20 minute sessions each. Two classrooms at one intervention school and 3 classrooms at another received this PSE intervention.
Fidelity and Dose Delivered. Three paraprofessional EFNEP educators with experience teaching community nutrition programs were responsible for delivering the PSE intervention curriculum and documenting the degree of program delivery. Given the lack of experience with the new PSE curriculum, all three EFNEP educators participated in two 2-hour curriculum training sessions and received an overview of the importance of process evaluation data collection, instruction in collecting process evaluation data, and instruction about completing the data collection forms and checklists as well. The data that the educators collected, reflected if lessons were delivered as intended and in a timely manner. Each EFNEP educator, responsible for two classrooms, assessed their own curriculum delivery by completing a fidelity checklist for each lesson. In addition, SNAP-Ed staff also observed each educator during three randomly selected lessons and documented program delivery using observation checklists to assess fidelity.
Dose Received. EFNEP educators also recorded their perception of the students' attentiveness and understanding during each lesson using the fidelity checklists.
In addition, dose received was evaluated by three take-home assignments throughout the study. The extent of assignment completion was evaluated by the average number of submitted assignments. Furthermore, each submitted assignment was scored using a rubric developed of each assignment to evaluate the students' learning.
Reach. EFNEP educators were also responsible for documenting the total number of students exposed at each lesson to assess the intervention's reach. In addition, the proportion of parent participation was evaluated by the number of submitted assignments which required parental input.
Perception of the Program. Data on the attitudes and perceptions of the intervention were collected by conducting one focus group discussion with EFNEP educators and two focus group discussions with six students from each intervention school.
Successes, barriers, and challenges to this intervention were also assessed through the handwritten notes and comments that EFNEP educators recorded using each lesson's fidelity checklist. In addition, semi-structured interviews were conducted with one school principal, three classroom health teachers, and an Aramark foodservice manager. Lastly, SNAP-Ed staff members attended the Pawtucket Wellness Committee's meetings and recorded the meeting minutes. These minutes were used to assess the committee's perceptions and acceptance of the program.

Instruments
Fidelity Checklists. Curriculum fidelity was primarily measured using checklists covering all lesson objectives, which were taken directly from each lesson plan. This instrument was developed for each lesson and it was completed by the EFNEP educator responsible for delivering the lesson. Items on the checklists reflected each lesson's objectives which educators completed by checking either "yes" or "no" to indicate which objectives were met. This instrument also documented student attendance, time spent preparing for each lesson, and time spent teaching. In addition, each checklist was supplemented with a survey assessing student attentiveness and understanding of the lesson. Educators could assess this by indicating the degree of attentiveness on a scale of 1 (not attentive at all) to 5 (very attentive) and understanding on a scale of 1 (did not understand) to 5 (understood everything). Space was also provided for educators to write notes and comments for each of their assessments.
Observation Checklists. Checklists were also developed for each of the three lessons SNAP-Ed staff observed throughout the intervention. This instrument documented fidelity of program delivery as well as objective data pertaining to the curriculum and student participation for each of the lessons observed. In addition, space was provided to record comments or suggestions for future implementation of the program.

Rubrics.
Rubrics were created to evaluate each of the three take-home assignments.
These rubrics evaluated whether students were successful in understanding lesson and/or activity objectives. Each rubric contained specific criteria for each assignment.
One SNAP-Ed staff member scored each submitted assignment by checking off "yes" or "no" to indicate if the assignment's criteria was met.
Focus groups. All focus groups with students and EFNEP educators were conducted with the assistance of focus group guides. These guides were developed based on previously tested focus group questions used in other SNAP-Ed interventions and were reviewed and edited by a SNAP-Ed staff member with prior focus group experience. The student focus group questions were piloted with five 5 th -grade students in a non-participatory school in Providence, Rhode Island.
Semi-structured interviews. Semi-structured interviews with one school principal, three health teachers, a foodservice manager, and members of the Pawtucket Wellness Committee were carried out at the intervention's conclusion with the use of interview guides. All interview questions were reviewed and edited by a SNAP-Ed staff member with previous interviewing experience.
Meeting minutes. Throughout the intervention year, SNAP-Ed staff attended the Pawtucket Wellness Committee meetings and were tasked with recording the meeting's minutes. These minutes were used to track any policy changes that took place as a result of the EMPOWER intervention.

Hypotheses
Hypothesis 1: Average fidelity and dose delivered of the EMPOWER intervention will be 80% as measured by educator self-reporting checklists and observation checklists.

Hypothesis 2:
Average student engagement and understanding assessed by educator checklists will be ≥80% and average student engagement and participation assessed by completion of take-home assignments will be ≥75%.

Hypothesis 3:
Average reach measured by the proportion of students participating in the EMPOWER intervention, as measured by student attendance per lesson will be ≥80%.

Hypothesis 4:
Students, school staff, and educators will evaluate the program positively as measured through focus groups and interviews.

Analysis
Quantitative data from each self-reported fidelity checklist, observation checklists, and grading rubrics were transferred to Microsoft Excel, which was used to analyze descriptive results (via averages and percent values). All handwritten comments from fidelity and observation checklists were typed onto a structured template. Focus group and interview responses were recorded via a note-taker. All responses were typed and reviewed with the note-taker to discuss initial finding and impressions. All checklist comments, focus groups, interviews, and meeting minutes were entered into NVivo11 (NVivo qualitative data analysis software; QSR International). Codes were generated from topics and questions covered in all the interview and focus group guides and checklist templates, which were then thematically analyzed. 30 The emergent themes are illustrated in this manuscript by selected anonymous quotes which exemplify the data.

RESULTS
The overall findings for each component and its respective instruments can be found on  Reach. Table 3 also lists the attendance for each lesson. On average, 134 students (94%) from both intervention schools were exposed to all 10 lessons.
Perception of the Program. The following section presents the common theme findings for each lesson, reported by EFNEP educators. Subjective data were evaluated to detect common themes between all three EFNEP educators. Common themes were identified by word repetitions and/or words in context. The following findings are presented from most mentioned themes to least mentioned as shown on The most emergent theme from all fidelity checklists indicate that student participation and engagement in lessons was high throughout the intervention. Attentiveness was particularly high for games and activities which involved group work and interaction with other students. As these educators illustrate: More communication between intervention and school staff was then determined to be an integral part for intervention success.
The curriculum's wordiness was also found to be a common challenge for all educators. All educators felt that some of the content was rather dry and needed to be condensed and more modified.
Tried to memorize lessons and rewrote the lessons because they were wordy.

Curriculum was very wordy, it had lots of talking. (Educator 3)
Timing of lessons also seemed to be a struggle that all educators perceived throughout the intervention year. This issue was tied into the students' difficulty understanding several aspects of the curriculum. Lessons were delivered every other week, and educators believed that this may have contributed to the students' PSE knowledge. As shown on Figure 4, what students recalled doing and enjoying more were creating their own posters advertising either fruits, vegetables, or the winning recipe. In addition, discussing barriers to eating fruits and vegetables was the lesson that students at both schools remember enjoying talking about. Overall, it was the interactive games and activities that students particularly enjoyed. Taste

I asked mom to put fruits and vegetables in refrigerator where I can see them.
(School 1) Semi-Structured Interviews. The interviews with the three classroom health teachers, one principal, and one foodservice manager were followed using an interview guide with questions asking about any perceived barriers, successes, suggestions for change, and any effect if any that the intervention had on their students. Like EFNEP educators, health teachers mostly expressed similar findings. The following quotes illustrate the most common perceived barriers.
Once more, delivering lessons every other week proved to be a major struggle for students.

The program was delivered every 2 weeks and a lot of students forgot what they had
learned on the previous lesson. Timing was the hardest. (Health teacher 2, School 1) …students were confused since having the class every two weeks was confusing to them and I am not sure they got it on a day-to-day basis. (Health teacher 3, School 2) The miscommunication between intervention and school staff was also made apparent by health teachers and foodservice.
Felt like sometimes we were not on the same page and there was some miscommunication. There needs to be more re-capping with EFNEP director.

(Foodservice manager)
There was miscommunication with the art teacher and there was confusion as to who was teaching what. (Health teacher 3, School 2) In addition, health teachers also mentioned the wordiness of the lesson plans as being a challenge in engaging student participation and understanding.
Have more hands-on activities and less talking from the instructor… kids got bored with a lot of lecture. (Health teacher 2, School 1) Script was very wordy and not very user friendly. The curriculum was a little over their head. (Health teacher 3, School 2) The same as students, all school staff that participated in these interviews as well as foodservice agreed that the recipe testing and contest was the most successful part of this intervention. All of them felt that students particularly enjoyed this aspect of the project and expressed their desire to see this intervention being delivered again.
The students really enjoyed taste testing the recipes. It was nice to see a different program that the students really enjoyed getting involved in. I would love to see the same program again. (Health teacher 2, School 1) The students really liked coming up with their recipes and polling the whole school. I think this was a great program and I would like to see it again. (Health teacher 1,

School 1)
Both cafeterias were very excited and looked like the kids really enjoyed Recipe day.

(Foodservice manager)
The recipe contest was awesome and the kids really enjoyed the lessons. (Principal, Lastly, the most commonly mentioned theme that school staff mentioned as a result of this intervention was student empowerment. Most agreed that the lessons and activities increased their students' self-confidence in requesting the fruits and vegetables they want to see being offered more, in school and at home.

DISCUSSION
The purpose of this study was to conduct a comprehensive process evaluation of a school-based PSE change intervention called EMPOWER. The primary aim was to assess the intervention's fidelity, dose, and reach as well as it's perception by various stakeholders and staff. This comprehensive process evaluation followed the comprehensive guide described by Saunders et al. 11 and its results have been used to fine-tune the intervention. Overall, both students and school-staff reported liking the intervention. Fidelity, dose, and reach were high throughout the intervention as well.
However, as expected from process evaluations, this study found areas to improve for future implementation. Some of the key changes include reducing the length of the lessons, simplifying language, including more interactive learning, and increasing communication between researchers and school staff.
The results from the interviews, focus group responses and handwritten portion of the checklists revealed that the EMPOWER intervention was perceived in a highly positive manner. Similar to other school-based interventions, 17, 31 the hands-on activities which in this study included the recipe taste-testing, creation of promotional posters, polling on "Recipe Day", and lesson games proved to be the most popular aspects of the intervention. The students' self-confidence and empowerment to have a voice in their school community and family environment increased as a result of these activities, as illustrated in the semi-structured interviews with the classroom health teachers and student focus groups. Student engagement has been discussed in other studies. 16,23,26 Researchers from these studies agree that increasing student engagement is an integral piece in assuring an intervention's success. One of the ways of ensuring engagement is by incorporating activities such as the ones reported in this study, which encourage self-efficacy to make their own choices. 23 Another way is by also amending lessons with take-home assignments to reinforce the skills learned. 16,31 In this study, 83 out of 142 students (58%) submitted a recipe as part of the lesson #5 take-home assignment. Students submitted a fruit or vegetable-based recipe from home, to participate in a school-wide recipe contest. The winning recipe was then provided on the school lunch menu one day during the intervention. Data from the rubrics used to evaluate the recipes revealed that only 21% met 7 out of 7 criteria with an average 4.7/7 score. Most of the recipes failed to provide specific quantities, suggesting that basic cooking skills are deficient in this population. However, 70% of the recipes submitted met the fruit or vegetable-based criteria, which was the primary goal of the take-home activity. Around 70% of the remaining two take-home assignments met criteria. Return rates dropped to 39% for the second activity which involved making requests to parents for fruits and vegetables. The last assignment about creating persuasive messages to eat more fruits and vegetables increased to a 97% return rate. Another study found that return rates tend to diminish over time. 31 However, in this study, the first two assignments required involvement from parents, which could explain the lower submission rates. Writing a recipe required students to interview a parent or family member, while the making requests assignment required a parent signature. This suggests that involvement from parents may have been low. In addition, all three take-home assignments were only written in English. The Pawtucket School District has a high percentage of Hispanic families (31%), which could also explain the lower participation from parents in these activities. Nevertheless, it should be noted that almost three-fourths of the students submitted their take-home assignments, which show that those students understood the lesson and activity objectives. Similar to the Active for Life Year 5 (AFLY5) study, the aim of incorporating take-home assignments was to reinforce the learning covered in the lessons and also extend the reach to parents or other family members. 16 However, other studies have not comprehensively analyzed returned assignment scores.

The data from the fidelity and observation checklists presented in this manuscript
show that the EMPOWER curriculum was implemented with a high degree of fidelity.
An average curriculum fidelity of 97% was recorded in the self-reported fidelity checklists and 95.6% in the observation checklists with a 99% agreement between self-report and observations. Percent agreement was measured by calculating the difference between the self-reported fidelity and observations. Results of this study compare favorably to other school-based intervention studies that have also used selfreported curriculum fidelity measurements and observations. Davis  were completed throughout the CATCH study. These two studies, which have found a lack of correspondence in completion of activities between self-reports and observations, raise the question of the validity of the self-reporting instruments.
Additional research that examines observations of all curriculum lessons is needed.
The dose delivered compares positively to other studies; 100% of the EMPOWER lessons were taught in all six intervention classrooms. In studies such as AFLY5, 77% of the lessons were delivered 16 and Project Tomato which reported an average of 45% implementation. 17 The dose of CATCH at 86%, although good was over-reported by the school staff who delivered the intervention. 32 Helitzer et al. also reported that some school teachers were not following lessons entirely. 22 In these studies that had low implementation rates, lessons were delivered by school staff rather than research staff, which may explain their outcomes. The studies such as It's Your Move 26 16 The EMPOWER lessons were designed to be taught in conjunction to the FFVP curriculum, however the PSE-change lessons took more time than intended. This also may have influenced the lack of FFVP delivery at both intervention schools. Lastly, inciting involvement of the Pawtucket Wellness Committee during the intervention proved challenging in this study. One parent and student dyad were recruited and attend one of the Wellness Committee meetings, however engagement from the committee itself was low. This could have been due to the recent creation of this Wellness Committee, whose recent creation unfortunately did not coincide well with this study.

Limitations
The fidelity checklists were completed by EFNEP educators and relied solely on self-report. Educators were observed three times throughout the intervention period. There was a 99% agreement between the self-reported fidelity and the observations. However, like many previous studies, these results should always be interpreted with caution. This has implications for future implementation at other schools; more observations by research staff may add more comprehensive data and reliability of the results. In addition, interviews and focus group responses were not audio recorded and transcribed. This decision was made to encourage student participation and a moderator and a note-taker were present at all focus groups and comprehensive notes were taken. Yet, findings also need to be approached with caution. Another limitation of both the semi-structured interviews and focus groups is that teachers and students might also be inclined to give socially desirable answers.
This could in turn lead to overestimation of the effects and perceptions of the intervention.

Fidelity
To what extent were each of the program's lessons implemented as planned?

Dose Delivered
Were all intervention components delivered as planned?
• Fidelity checklists Was feedback provided to the Wellness Committee?
• Copies of Wellness Committee meeting minutes

Dose Received
To what extent did students engage in lesson activities?

• Fidelity checklists
To what extent did the students complete assignments?
• Number of submitted take-home assignments Did the students learn?

Reach
Was the curriculum delivered to at least 80% of fifth grade students?
• Student attendance What proportion of parents participated in the intervention?
• Copies of family recipes • Graded rubrics

Perception of Program
How did the students react to the intervention?
• Student focus groups (2) How did educators and school staff react to the intervention?
• EFNEP focus group • Fidelity checklist notes • Interviews (4) Did the students improve attitudes about fruits and vegetables and feel empowered to change fruit and vegetable options?
• Student focus groups     including how the intervention was designed, how successful it was at delivering its different components as they were originally planned, and how much audience participated and/or were exposed to the intervention. 1 These elements are what process studies aim to evaluate: to enhance the understanding of intervention results.
There are differing methods by which PSE interventions are evaluated, therefore, details of the methods and instruments used to document the process will be examined.

Childhood Obesity
The prevalence of childhood obesity is a major health problem in the United States. It has been documented that the prevalence of elementary-school children between 6 and 11 years of age with obesity (body mass index at or above the 95 th percentile for age) has increased from 4.2% in 1963-1965 to 18.0% in 2009-2010 2 and since then has remained fairly stable. 3 Moreover, lower-income and ethnic populations are at a greater risk and have the highest rates of obesity. 2,4 Overweight (body mass index at or above the 85 th percentile and below the 95 th percentile for age) and obese children pose a major public health concern since many children who are overweight or obese maintain their obesity as adults. This in turn, leads to related comorbidities such as diabetes, heart disease, high blood pressure, high cholesterol, stroke, some cancers, arthritis, and sleep-disordered breathing. 5 Multiple factors influence obesity. Not only are genetics a cause, but the environment where we live, work, and play is also a major determinant of our dietary and physical activity habits. 6 In addition, evidence suggests that community-level policies that affect local food environments, may also be contributing either positively or negatively to the obesity epidemic. 6

Policy, Systems, and Environmental Change
Strategies to reduce the prevalence of obesity involve changing individual health behaviors. 8,9 However, public health professionals are now also targeting the policies, systems, and environments (PSEs) that support this behavior change. 9 A sedentary lifestyle and increased intake of unhealthy foods and beverages are more commonly found in community areas where there is a decreased access to healthy foods, increased exposure to advertising and availability of fast food, and a lack of access to safe recreational areas that promote physical activity. 8

School-based PSE Interventions
Given that on average, a child obtains about 26% of their total energy intake during the school day, PSE change interventions in schools have been deemed as top priorities in the battle against childhood obesity by both the American Academy of Pediatrics and the Institute of Medicine. [12][13][14][15] In addition, schools are the only setting where many children are gathered and can be provided with opportunities to receive education on a healthy lifestyle. 14 The aim of PSE change interventions in schools, unlike individual nutrition education interventions, is to change the school setting by targeting system-wide policy and environmental factors so that the entire school community (students, student's families, and school staff) will be positively affected and encouraged on a daily basis to make healthier choices. 12 16 Other previous studies that have examined PSE changes in middle schools and how they affect food consumption in students have found mixed results. [17][18][19] A two- year intervention by Sallis et al. found that the policy and environmental changes they implemented were effective in improving physical activity but were not successful at reducing total and saturated fat intake from all school food sources including cafeteria, a la carte foods, school stores, and bag lunches. 17  Changes included increasing fresh fruit and vegetable availability and decreasing high fat snack items and sweetened beverages in cafeterias and vending machines. One of six middle schools did not attain the 75% goal achievement, but overall the researchers found that in the short-term of six weeks, the foodservice changes were acceptable to students and school staff. 20 However, changes to the vending machines proved the most difficult due to vendor contracts and sources of revenue to the school. 20 Generalizability of this intervention is limited due to its short duration and lack of data on actual student dietary intake.
Although these studies provide some evidence of the effectiveness of PSE change interventions in schools, most have encountered similar issues along the way. 14, 17-20 These issues include, difficulty implementing school food changes due to financial constraints (vending machine contracts, fundraising, etc.), failure to control unhealthy foods brought from home, lack of integration into daily school activities due to delivery of intervention solely from research staff, and difficulty of implementation within the context of standardized academic performance testing. 14 These barriers and challenges have been clearly documented in the literature due to the investment of many public health professionals in building an evidence base for the emerging study of PSE change. 21

Process Evaluation
In recent years, public health research has increasingly incorporated qualitative methods into their PSE change outcome studies due to the variability of program implementation and policy adoption, particularly in school and community settings. 1 Unlike outcome studies that seek to determine if an intervention was successful or unsuccessful, process evaluation studies are implemented to determine why and/or how such an intervention attained its respective results. 22 Evaluations such as these can also aid in demonstrating progress and effectiveness before actual outcome results are measured. 23 In addition, if an outcome study was unable to achieve positive results, process evaluation can aid in using the data collected throughout an intervention to identify potential causes and suggest how that unsuccessful intervention could be modified and improved upon, instead of relying on mild speculation in order to explain why and how. 22 Process evaluations gather data on the social processes involved in the delivery and reception of the intervention. They frequently entail mixed methods involving questionnaires, semi-structured interviews, focus groups, direct observation, and checklists. These different evaluation components provide data to describe how a program was implemented, how well the activities delivered fit the original design (fidelity), to whom was the intervention delivered to (dose), the extent of the target population that was reached (reach), and any other external factors that may influence the intervention's effects. 22 quantifying actual effects of the intervention to the outcomes. 23 Therefore, process evaluation plans and designs tend to typically evolve over the course of an intervention, to fit stakeholder priorities and program delivery. 21

Process Evaluations of School-based PSE Change Studies
One of the first school-based process evaluation studies was one within the Child and Adolescent Trial for Cardiovascular Health (CATCH) which targeted dietary behaviors, physical activity, and smoking through PSE changes in four core programs, including school foodservice, physical education, classroom curricula, and parental involvement. 25 An extensive amount of process data was gathered for each of the four programs during the three-year intervention period, to provide insight of how the CATCH program was implemented and how they successfully implemented the intended PSE changes among the 56 intervention schools. The process measures used specifically for the process evaluation of the classroom curricula were to document teacher exposure to the curriculum training sessions, how much of the curriculum was implemented, to what degree it was implemented as designed, and the barriers to implementation. 26 Teachers were administered questionnaires which examined attendance at training sessions and perceptions from both training sessions and the curriculum itself along with questions targeting self-efficacy of delivering the curriculum. Dose and fidelity of curriculum implementation was measured using selfreported weekly checklists and empirical observations of selected class sessions conducted by research staff. Interviews with teachers were also conducted after the program was concluded, to obtain feedback on individual sessions and the CATCH program as a whole. The data that was collected from all program components was then successfully used to describe implementation of the program for quality and monitoring purposes and also helped explain the program's effects. 26 The data collected revealed that 100% of teachers involved in the intervention attended all training sessions and the fidelity of implementation was referred to as excellent (more than 90%). There was also high compliance in completing the weekly checklists which revealed high teacher satisfaction. Interviews exposed teacher uncomfortableness with being observed, however during interviews teachers did acknowledge the interventions impact on their students' behaviors, and the most common barrier encountered was the length of each lesson.
Subsequently, more studies began incorporating process evaluations in their research studies following guidance from innovative studies such as CATCH. The process evaluation of an obesity-prevention trial for American Indian schoolchildren by Helitzer et al. examined whether and how the intervention was implemented during the pilot phase. 27 This study described the development and pilot testing of the process evaluation instruments, how these instruments were selected for use on the full-scale trial, and provided information on how the process evaluation results were used to fine-tune the program overall. 27 The research group also developed an extensive data collection method, including 27 sets of instruments involving checklists, attendance logs, self-administered evaluation forms, individual lesson feedback from teachers, structured interviews, surveys for student feedback, surveys for student exposure questions, observation checklists, and meeting minutes. 27 Results from the process evaluation of the pilot study were used to monitor implementation of all the study components and provide input and fine-tune the components and revealed the need for more precise instruments. 27 Through direct observation of lessons, the research group found that most teachers completed the checklists and evaluation forms and gave above average rating to the 12-lesson curriculum. Teacher satisfaction increased throughout the intervention. However, they found that their open-ended evaluation questions were not very clear to the teachers and therefore provided less useful information. This finding helped improve the evaluation forms. The observations also revealed that teachers were not delivering the curriculum as planned by omitting several parts of lessons and several activities. This indicated a need for more emphasis on the importance of maintaining curriculum fidelity during teacher training sessions.
Interviews with teachers and school-staff revealed high satisfaction with the intervention, however several issues were discovered such as lesson duration, lack of training in how to control children during the PE component, and lack of curriculum flexibility. Student exposure was evaluated by administering questionnaires with 15 exposure questions. The data showed that more than 80% of intervention students reported exposures to 7 out of 15 items, however less than 70% reported exposure to 5 of 15 items. 27 These results suggested to the researchers the need for more specificity in the questionnaires since several items described activities that could apply to any elementary school curriculum. 27 The Gimme 5 Fruit and Vegetables for Fun and Health was a multicomponent intervention, which included 12 lessons, designed to increase fruit, 100% fruit juice, and vegetables in fourth-and fifth-grade students. 28 The process evaluation of this intervention by Davis and colleagues, assessed fidelity of implementation, reach, and use of the intervention materials, which included teacher training sessions, curriculum delivery, family participation in activities, attendance to grocery store activities, and availability and accessibility of fruits and vegetables at home. 29 Data was collected with the use of observations (at least once per teacher), self-reported checklists, and interviews. Thirty-three teachers in fourth-grade (44 total observations) and 36 teachers in the fifth-grade (59 total observations) were observed and it was found that about half of the curriculum activities were completed. In contrast, teachers reported completing 90% of curriculum activities which raised the question of self-reported bias. 29 Ninety-five percent of participating teachers participated in the training sessions. In addition, 95% completed the curriculum checklists, however no reliability was determined for this measure. Eighty-five percent of teachers rated the lessons as excellent to outstanding (4.6 to 4.8 on a 5-point scale). 29 Common barriers that were exposed included length of lessons, dependability of parent participation, and repetitiveness of material. Interviews were only conducted with fifth-grade teachers. Thirteen to 16 parents were interviewed on the telephone, and were asked questions regarding homework and any materials brought home, participation in parent and grocery store activities, and fruit and vegetable accessibility at home. Five percent of parents reported receiving all 6 newsletters sent home (56% reported receiving between 3 and 4), 87% participated in homework activities,10% reported attending grocery store activities, and fruit availability and accessibility at home was found to have increased significantly (p=0.02 and p=0.003 respectively) however the same was not found for vegetables (p=0.14). Similarly, other studies have also found challenges in extending program reach beyond the student community to increase knowledge and skills to parents. 30, 31 The process evaluation for Project Tomato, a randomized controlled trial of a well received by all three groups and the most commonly accepted items included hands-on activities such as games and recipe taste-testing. However, the main barrier that was found was preparation time, lack of training, and a seemingly labor-intensive intervention. In conclusion, the researchers did not find a positive association between the intervention and the children's eating behavior and process data was able to expose a poorly implemented intervention, similar to another study by Campbell et al. 32,33 Another study from the United Kingdom called Food for Fitness, was a multicomponent program as well that was conducted in elementary and middle schools. 34 In addition, trained community nutrition assistants delivered this intervention. The process evaluation, conducted by Middleton et al. used thirteen semi-structured interviews and two focus groups with stakeholders throughout the intervention which included nine health professionals, ten school teachers, and three senior health officials. These qualitative evaluation methods focused on examining how the program was received by the stakeholders (such as its impact on the students) and how the program was delivered (such as the quality, organization, and availability of the service). These measures aimed at going beyond the "yes/no" and "how much" questions, by instead focusing on qualitative inquiry that would provide more depth by drawing out more understanding and perceptions of the program. The researchers analyzed the transcribed data, coded common themes, and categorized them as either belonging to program receipt or program delivery. The results showed that school teachers perceived the program as a good service, while the health professionals and senior health officials involved in the program perceived it as vital or essential to changing students' health behaviors. However, several program delivery issues were exposed. These were issues concerning program planning, the limited size of the intervention, and difficulty sustaining long term nutritional goals at the schools. 34 Volpe et al. conducted the process evaluation of the HEALTHY study, a large multicenter trial to decrease the risk factors of type 2 diabetes mellitus in 21 middle schools by promoting physical activity and nutrition. 35 The aim of the HEALTHY study was to improve the quality of the foods and beverages offered to students by changing the total school food environment. Research dietitians and foodservice staff worked together to make environmental changes and organize activities that encouraged students to try new foods at breakfast and lunch. Process Forming strong communication between foodservice staff and dietitians was a common theme among interviews and was then considered of topmost importance if the nutrition goals were to be met. As with other studies previously mentioned, the most challenging barriers were costs, as well as availability of foods, and student acceptance.

Conclusion
Childhood obesity rates in the United States have plateaued in recent years. 2 However, it still continues to be a major public health concern particularly in lowincome and ethnically diverse communities. 2,3 There are several known factors that have influenced this epidemic, and the environment in which we live, work, and play has been identified as a key contributor. 6 Policy, systems, and environmental change strategies which aim at modifying said environment are increasingly being implemented in many community settings, with particular interest in schools. [12][13][14][15] These PSE change strategies aim at changing health behaviors at a population level, which are not determined to have more impact than interventions at the individual level. 8 However, due to the varying success of many school-based PSE change interventions, studies are including more process evaluations to help explain their final outcomes. [17][18][19][20][21] Process evaluations are implemented to determine why an intervention was successful or not, and can also be used to demonstrate an intervention's progress and effectiveness before outcomes are measured. [21][22][23][24] They gather data on the social processes involved in the delivery and reception of an intervention by measuring its fidelity, dose, and reach. 24  improve the fruit and vegetables choices in their homes and school and get more people to eat fruits and vegetables. 9. Talked about the ways the students will make these changes: mentioned the projects the students will be involved in 10. Went through "Think About Fruits and Vegetables in Your Environment" activity with teams of 3-4 students and had 1 reporter from each team 11. Explained why students will be writing letter to the Wellness Committee with common barriers to eating fruits and vegetables 12. Drafted letter using top responses from ""Think About Fruits and Vegetables in Your Environment" activity and explained that the students will be signing it 13. Opened invitation for 1 student and their parent(s) to join Wellness  Asked students to answer "What are some of your barriers to eating fruits and vegetables" 4. Identified top barrier to eating fruits and vegetables 5. Lead students to brainstorm solutions or strategies for overcoming their top barrier 6. Drafted the final letter to the Wellness Committee including their barriers and solutions 7. Read the final draft of the letter to the class 8. Asked the students if anything else should be added to the letter 9. Passed the signature sheet around the classroom for students to sign their name 10. Explained what an "Environmental Scan" is, deconstructing the words "environment" and "what it is to scan" 11. Explained that 2 students from each school and their parent(s) will join the Pawtucket Wellness Committee 12. Lead students in reflecting on what they learned on today's lesson 13. Asked students if they have ever followed a recipe 14. Introduced next lesson: the importance of following a recipe  Discussed what a recipe is and why they are important to have and read before cooking 2. Discussed "Curly Kale Slaw" recipe using props and materials (measuring spoons + cups) 3. Explained the descriptive words "minced" and "chop" 4. Explained that students need to ask about precise amounts during their interviews with the help of their measuring spoons and cups 5. Asked students if the directions for the recipe were easily understood 6. Explained that students need to ask about detailed directions during their interviews 7. Explained and completed the recipe card activity 8. Asked the class about what they learned and the importance of having complete and accurate recipes 9. Introduced next week's activity and discussed what "role-playing" is Please check "yes" or "no" to indicate if each of the major objective, activity or point to make below was covered when the session was taught.
Yes No

1.
Discussed what a role-play and an interview is and how the students will use them for their activity 2. Explained that the recipes that the students will be interviewing about need to follow certain guidelines including a fruit and/or vegetables as the main ingredient 3. Gave an example of how carrot cake and vegetable pizza do not contain a vegetable as the main ingredient 4. Gave an example of how stir-fried garlic broccoli does have a vegetable as the main ingredient 5. Discussed how recipes need to have step-by-step directions 6. Explained that students will have a script for their role-play activity and interviews at home and demonstrated the activity with the classroom teacher 7. Asked students to verify if their recipes followed all guidelines on the Recipe Checklist 8. Instructed students to take home the interview script and recipe card to complete their interviews at home 9. Explained the purpose of the Parent Newsletter and instructed the students to write-in their "project due date" 10. Introduced next week's activity by discussing what a request is and how to make one for fruits and vegetables Please check "yes" or "no" to indicate if each of the major objective, activity or point to make below was covered when the session was taught.
Yes No 1. Allowed 1 or 2 students to share their recipe with the class 2. Explained that recipes from each classroom will be taste tested and students will vote for a winning recipe to be featured on the school lunch menu 3. Discussed what it is to make a request 4. Discussed why parents don't want to buy fruits and vegetables that go to waste (because their kids don't eat them) 5. Discussed solution to barrier by asking parents what students like instead of what they don't like 6. Discussed how to make a request by: noticing something you like>making a positive statement>making a request 7. Gave examples of a request and had students identify the "positive statement" and the "request" 8. Explained and went through the directions for the "Making Requests" activity, emphasizing the need for it to be related to fruits and vegetables 9. Asked the students to take the worksheet home and have parents sign 10. Discussed what a poll is and explained next week's recipe taste test and poll taking activity 11. Explained how the most voted recipe from all 5 th grade classrooms will be served to the entire school at lunch time Explained that students will be writing slogans for fruits and vegetables 5. Read through "Top 10 Reasons to Eat Fruits and Vegetables" handout 6. Explained that every advertisement has a picture with and that students will be creating posters for each of their slogans 7. Explained the "Writing Slogans" group activity and showed an example of a slogan and a sketch poster 8. Explained that students will be creating their posters in art class Please check "yes" or "no" to indicate if each of the major objective, activity or point to make below was covered when the session was taught.

Yes No
1. Introduced morning announcement project 2. Discussed why posters and announcement are important to reach an audience 3. Explained how advertising messages try to get people to do or buy things 4. Discussed the students' purpose and importance of creating their ads (to get people to eat more fruits and vegetables) 5. Introduced writing a persuasive message activity by discussing the 3 messaging strategies (feel good, information and build trust) 6. Went through examples of messages and had students decide which type of message each was 7. Instructed the students to write their own persuasive messages using the messaging strategies 8. Went through Creating Messages Guide handout and instructed students to use for their messages 9. Allowed each group to share one message they created 10. Prompted students to share one thing they learned about advertising from doing the activity 11. Explained that posters and slogans are up on the school walls and that morning announcements will be read next week 12. Introduced poll taking practice for next lesson Participants demonstrated a sense of understanding of the lesson.

2.
Used at least one of the messaging strategies (appealing to emotions, giving information, or build trust) each question I will give you some time to answer aloud. You don't have to speak in order. If you want to answer a question, you can, just be sure not to talk over another student. You do not have to answer a question if you don't want to but just so you know, there are no right or wrong answers and you will not be graded on anything you say. We are only asking you to be as honest as possible so you can help us improve our program. Do you know what it means to be honest?
Do you have any questions about what I explained before we get started?

Ice-Breaker
• Name tags • Markers Start by writing your names on these tags so we can get to know each other a little better.
Do you remember the two recipes that you voted for in your class? Which were they?
Let's go around the circle and say which recipe you voted for and why you liked it. Thank you so much for meeting with me today! All of your comments have been very helpful.

Notes Introduction
• Thank you • Purpose • Confidentiality • Duration • How the interview will be conducted • Opportunity for questions

Say,
Thank you so much for taking the time to meet with me today. I would like to talk to you about your experiences as educators in the URI Nutrition Program. As part of our program evaluation we are assessing program effectiveness and acceptability. What you have to say will help improve the program for future interventions.
_______ will be taking notes during this time just so I am sure to get it all down.
All responses will be kept confidential and will only be shared with the other research team members. Any information that's included in the final report will not identify you as the respondent. You do not have to answer a question if you don't want to and may end the interview at any time.
Do you have any questions before we get started?

Say,
Thank you so much for taking the time to talk with me today. My name is ________ and I would like to talk to you about your experiences participating in the EFNEPenhanced PSE Nutrition Program. As part of our program evaluation we are assessing program effectiveness and acceptability. What you have to say will help us improve our program for future interventions.
I will be recording the session because I don't want to miss any of your comments. However, I will also be taking notes during this time just so I am sure to get it all down. Because we're going to be recorded, I would just like to ask you to please be sure to speak up so that we don't miss any of your comments.
All responses will be kept confidential and will only be shared with the research team members. Any information that we include in our final reports will not identify you as the respondent. You do not have to answer a question if you don't want to and may end the interview at any time.
Do you have any questions before we get started?
Questions 1) What strategies or components from the curriculum would you recommend be discontinued?
2) Would you just get rid of this component or would you change/alter it?
3) What worked well? Please elaborate 4) What strategies or components from the curriculum would you recommend be sustained and/or expanded?

Notes Introduction
• Thank you • Your name • Purpose • Confidentiality • Duration • How the interview will be conducted • Opportunity for questions

Say,
Thank you so much for taking the time to meet/talk with me today. My name is ________ and I would like to talk to you about your experiences participating in the EFNEP-enhanced PSE Nutrition Program. As part of our program evaluation we are assessing program effectiveness and acceptability. What you say will help us improve our program for future interventions.
I will be recording the session because I don't want to miss any of your comments. However, I will also be taking notes during this time just so I am sure to get it all down. Because we're going to be recorded, I would just like to ask you to please be sure to speak up so that we don't miss any of your comments.
All responses will be kept confidential and will only be shared with the research team members. Any information that we include in our final reports will not identify you as the respondent. You do not have to answer a question if you don't want to and may end the interview at any time.
Do you have any questions before we get started? Is there anything else that they would like to add? I'll be analyzing the information you and others gave me and submitting a final report. I'll be happy to send you a copy to review at that time, if you are interested.
Thank you so much for your time to meet me today.