SNAP-ED EVALUATION FRAMEWORK Practitioner Stories As an accompaniment to the SNAP-Ed Evaluation Framework Interpretive Guide, the Food and Nutrition Service (FNS), in partnership with the National Collaborative on Childhood Obesity Research and the Association of SNAP Nutrition Education Administrators, produced this collection of vignettes. The stories describe how SNAP-Ed agencies have used the evaluation framework, as told by practitioners themselves. Since there is no single way to apply the framework, it is important to make the framework meet your agency’s needs. This document presents exemplars that are ordered strategically to illustrate comprehensive approaches through broad application of the framework followed by narratives that reference specific aspects or indicators. Overall, the vignettes focus on the creation of statewide evaluation and reporting systems, explain how the evaluation framework informed the development of state and local work plans, and highlight ways that the SNAP-Ed Evaluation Framework helps to communicate program evaluation results to stakeholders. FNS intends to grow this collection over time as more states incorporate the evaluation framework into their State SNAP-Ed Plans. For consideration, please share your brief vignettes (approximately 150 words) on how the evaluation framework has inspired your state or agency with your Regional SNAP-Ed Coordinator. 1 USDA is an equal opportunity provider, employer, and lender. AGENCY TYPE(S) FOOD AND NUTRITION SERVICE REGION MID-ATLANTIC (MA), MIDWEST (MW), MOUNTAIN PLAINS (MP), WESTERN (W), NORTHEAST (NE), SOUTHEAST (SE), SOUTHWEST (SW) SNAP-Ed Evaluation Framework Practitioner Story SNAP University Health Non- MA MW MP NE SE SW W Agency Department Profit Kansas – Producing PEARS, A Data Tracking System for X X the Mountain Plains Region Arizona – Using the Framework to Inform Arizona’s x x x Statewide Evaluation Plan California – The Potential for Shared Statewide Metrics x x x x x Maine – Creating a Standardized Program Evaluation x x x System in Maine Missouri – Reinforcing Multi-Sector Partnerships Using the x x SNAP-Ed Evaluation Framework’s Indicators Michigan – PE-Nut™: A Whole-School Intervention for x x Healthy Eating and Physical Activity Michigan – Applying the SNAP-Ed Evaluation Framework to x x Michigan’s Statewide Social Marketing Campaign Maryland -- Employing Specific Indicators to Evaluate x x Maryland’s Text2BHealthy Program Georgia – Measuring Physical Fitness in Georgia’s x x x x x Schools—An Illustration of the Impact of One Indicator in the SNAP-Ed Evaluation Framework Tennessee - Scoring Coalitions using the SNAP- Ed x x Evaluation Framework’s Multi-Sector Partnerships Indicator Alaska – Measuring food security in remote villages x x x California – SNAP-Ed Healthy Behavior Initiative Leads, x x x Unexpectedly, to the Distinguished After School Health Recognition Program 2 2 PRODUCING PEARS, A DATA TRACKING SYSTEM FOR THE MOUNTAIN PLAINS REGION Coordinated by Kansas State Research and Extension, Kansas SNAP-Ed’s nutrition education centers around the following four SNAP-Ed Evaluation Framework indicators: MyPlate Behavioral Change (indicator MT1), Food Resource Management (indicator MT2), Physical Activity and Reduced Sedentary Behaviors (indicator MT3), and Food Safety (indicator MT4). The state uses the Program Evaluation and Reporting System (PEARS) to gather evaluation data. Using PEARS, Kansas SNAP-Ed professionals can select or build evaluation instruments specific to the needs of each program they deliver, enter program information and response data, and access various reporting features. Moving forward, the system will also track the assessment of policy, system, and environmental change (PSE) efforts. Details about PSE progress, implementation, and impact at both the individual and environmental levels are useful for reporting to internal and external stakeholders. Using PEARS has allowed Kansas to have consistent data tied to program objectives. As a result, Kansas is in a position to more effectively measure and report program impact and is now partnering with neighboring states to make PEARS available as a region-wide data tracking system for the Mountain Plains Region. Screenshot of the PEARS Login Page 3 3 USING THE FRAMEWORK TO INFORM ARIZONA’S STATEWIDE EVALUATION PLAN In 2015, in preparation for a new grant cycle, the Arizona Department of Health Services (ADHS) sought a comprehensive, statewide evaluation plan that incorporated both direct education and policy, systems, and environmental change (PSE) approaches. The external evaluator for Arizona’s SNAP-Ed program, the University of Arizona Evaluation Team, turned to the Western Region Office (WRO) SNAP-Ed Evaluation Framework (now the national SNAP-Ed Evaluation Framework) as a guide. Because the Social-Ecological Model was foundational to Arizona’s SNAP-Ed program delivery model—and therefore the evaluation of that programming—the framework provided an ideal scaffold for a state-level one. What has emerged as the Arizona SNAP-Ed Evaluation Framework combines Arizona’s 16-strategy programming with the national SNAP-Ed Evaluation Framework. Individual-level indicators from the SNAP-Ed Evaluation Framework assess the state’s direct education strategy; environmental settings indicators measure PSE work that is site-based or with single partners; and sectors of influence indicators evaluate multi-sector partnerships and coalitions. Arizona SNAP-Ed Evaluation Framework - Excerpt of a Food Systems Strategy 4 THE POTENTIAL FOR SHARED STATEWIDE METRICS California has five SNAP-Ed State Implementing Agencies (SIAs), each of which has its own Local Implementing Agencies (LIAs) for reaching the SNAP-Ed population with education and obesity prevention interventions. The California Department of Social Services (CDSS) serves an oversight function and funds 6 county welfare departments. The California Department of Public Health (CDPH) funds 59 local health departments. The California Department of Aging has 19 area agencies on aging. UC CalFresh administers SNAP-Ed through UC Cooperative Extension in 31 counties. Catholic Charities of California has 11 local agencies across 24 counties. At the county-level, the LIAs jointly develop a single SNAP-Ed integrated work plan. Four of the SIAs use the same Food Behavior Checklist to evaluate behavior change among SNAP- Ed adult participants in series classes. In federal fiscal year (FFY) 2015, these four SIAs found statistically significant improvement in healthy eating behaviors in indicator MT1c (Ate more than one kind of fruit) and MT1d (Ate more than one kind of vegetable) across more than 3,000 matched participant surveys. The five SIAs have developed common SNAP-Ed goals and objectives for FFY 2017–2019 and are currently working on a joint evaluation plan. The SNAP-Ed Evaluation Framework will be useful for identifying shared metrics that can tell a more unified story across the California SNAP-Ed SIAs. Map of California’s 58 Counties 5 CREATING A STANDARDIZED PROGRAM EVALUATION SYSTEM IN MAINE The Maine Department of Health and Human Services contracts with the University of New England to deliver SNAP-Ed through 23 community-based coalitions. Maine’s nutrition educators enhance direct education with obesity prevention policy, systems, and environmental change (PSE) strategies. Maine developed a tracking tool that facilitates planning and collects data on short- and medium-term indicators. Specifically, the tool captures activities that directly relate to elements in the SNAP-Ed Evaluation Framework, namely readiness, champions, partnerships, and nutrition supports adopted. The extent to which PSE efforts reach the SNAP-Ed eligible population can be analyzed by setting and geography. In 2015, 40 nutrition educators implemented strategies across Maine to achieve the following: implement 14 community and 6 school gardens; increase access to federal food programs for youth in 20 sites; and improve wellness policies in 13 schools and 8 child care centers. In total, 62 schools, 6 afterschool programs, 12 child care centers, and 46 community organizations (e.g., public housing and food pantries) were targeted. Tracking progress using nationally established performance measures allows Maine to monitor program effectiveness in a clear, consistent manner. The data inform quality improvement efforts and ensure local-level success and national relevance. Edible Main Street in Norway, Maine School Gardens in Lewiston, Maine 6 REINFORCING MULTI-SECTOR PARTNERSHIPS USING THE SNAP-ED EVALUATION FRAMEWORK’S INDICATORS The Missouri Council for Activity and Nutrition (MOCAN) is an example of a multi- sector partnership consisting of representatives from more than 50 statewide and local agencies, institutions, organizations, other coalitions, and individuals. Together, MOCAN partners engage in coordinated activities to advance the goals and objectives of the statewide SNAP-Ed Plan, Preventing Obesity and Other Chronic Diseases: Missouri’s Nutrition and Physical Activity Plan. Many of the partners focus on specific audiences, such as SNAP-Ed eligible audiences, but the breadth of MOCAN’s partnering agencies allows consistent messaging and coordinated delivery strategies to occur across agency boundaries. MOCAN is exploring how the SNAP-Ed Evaluation Framework can be used to communicate the collective impact across its work groups by focusing on community issues such as food systems, which relates to the MT8 and LT12 indicators. Others issues addressed, with their respective indicators in parentheses, are built environments (MT5/LT5 and MT6/LT6), social marketing (MT12), health care (MT11), and government policy (MT7). The long-term goal of these efforts is to improve Missouri’s status on the population results indicators (R1-R11). More information on MOCAN can be found at http://extension.missouri.edu/mocan/. 7 PE-NUT™: A WHOLE-SCHOOL INTERVENTION FOR HEALTHY EATING AND PHYSICAL ACTIVITY Developed by the Michigan Fitness Foundation, PE-Nut™ is a multi-component, whole-school intervention that focuses on health behavior changes at both the classroom and school levels as well as in the home. In doing so, the program provides both teacher and student input on those changes. Each of the four program components was developed to teach children why and how to make healthy food choices, the importance of physical activity, and the skills necessary to be active for life. One of the components, Healthy Classrooms, Healthy Schools, focuses “At first, when I asked for healthy on transforming the classroom food at our room holiday parties, it environment through healthy snack policies; healthy school meal and was met with resistance. As we went vending machine options; visual cues through the program, the last party and messaging in the classroom and had the healthiest food of all.” throughout the school; role modeling (Words of a participating teacher) by teachers and students; classroom policies for not using food as rewards; and healthy classroom party policies. These serve as examples that collectively result in positive shifts in social norms that are prevalent in schools. Program outcome evaluations have built the evidence base with indicators that match those in the SNAP-Ed Evaluation Framework to establish the PE-Nut™ program as a best practice intervention for increasing student consumption of healthy foods and physical activity. The success of PE-Nut™ comes from its multi-level approach to interventions. More specifically, the activities not only teach nutrition concepts and promote physical activity, but they also offer opportunities for students to apply what they learn by trying healthy foods and by being physically active throughout the day. Sending the messages into the home helps to ensure a long-lasting impact. PE-Nut™ Includes Multiple Components to Support Behavior Changes in Classrooms, Schools, and at Home 8 APPLYING THE SNAP-ED EVALUATION FRAMEWORK TO MICHIGAN’S STATEWIDE SOCIAL MARKETING CAMPAIGN Two Campaign Billboards Featuring USDA’s Core Messages Developed by the Michigan Fitness Foundation (MFF), They Learn From Watching You (TLFWY) is Michigan’s two-pronged, multi-level statewide social marketing campaign. It is based on the USDA’s core messages to increase fruit and vegetable consumption (Eat more fruits and veggies and they will too) and physical activity (Be active and your kids will too) of children and their families. This practice-tested social marketing campaign uses multiple, overlapping channels that include social media/digital platforms, outdoor billboards, bus and truck wraps, print collateral, and public service announcements. On-the-ground components of TLFWY include Michigan Harvest of the Month (MiHOTM), which is modeled after California’s Harvest of the Month social marketing campaign. MiHOTM’s ready-to-go supplemental nutrition education materials can be integrated into core curricula and are based on the current Dietary Guidelines for Americans. All materials feature produce that is readily available and affordable in Michigan. As a result, MiHOTM has become widely popular in community, school, child care, worksite, retail, farmers market, health care, and emergency food settings. Via training on MiHOTM by MFF and its partners, teachers in the Michigan Department of Education (MDE) Fresh Fruit and Vegetable Program schools are implementing MiHOTM activities and using its resources throughout classrooms and cafeterias. This is part of an effort to coordinate the delivery of FFVP snacks with nutrition education to increase fruit and vegetable intake among children. Another example of partnerships and social marketing integration is Michigan’s Produce for Pantries, a collaboration between MFF, MDE’s Food Distribution Unit, local food producers, emergency food providers, and communities of faith. The pilot initiative promotes increased fruit and vegetable consumption among emergency food recipients through increased access to fresh produce (environmental change); onsite TLFWY social marketing (environmental change/social marketing); and onsite MiHOTM nutrition education (systems change/direct and indirect nutrition education). Without a doubt, the SNAP-Ed Evaluation Framework’s MT12 indicator will be a great resource for evaluating Michigan’s campaign. 9 EMPLOYING SPECIFIC INDICATORS TO EVALUATE MARYLAND’S TEXT2BHEALTHY PROGRAM The University of Maryland SNAP-Ed program developed Text2BHealthy, a text message–based program for parents that reinforces nutrition information and behaviors their children are learning in school SNAP-Ed programs. On average, Text2BHealthy is implemented in 15 Maryland elementary schools and reaches over 2,000 parents annually. Parents who participate in Text2BHealthy are surveyed before and after the program about grocery shopping habits; familial eating and physical activity behaviors; and home nutrition environment characteristics. Real- time evaluation is also done through texted questions, which assess parents’ responses to previous educational messages. The data collection is used to provide information on program relevance and applicability as well as to explore whether text messaging, alone and in combination with in-school SNAP-Ed youth programming, is associated with improved health outcomes among participating families. The results have informed message content and delivery and have also encouraged program expansion and replication by other agencies. The evaluation of Text2BHealthy is part of a comprehensive parent-wide evaluation plan that capitalizes on indicators offered by the SNAP-Ed Evaluation Framework. Specifically, it assesses both individual behavior changes (indicators MT1 and MT3) and organizational adoption of healthy supports (indicators MT5 and MT6). 97% of Participants Reported Healthy Behaviors 10