Detailed Guide to Evaluating Financial Education Programmes * * With the support of the Russian/World Bank/OECD Trust Fund Table of Contents “Are we making a difference?” p.3 What is evaluation? p.3 Why are we doing this?” - Understanding your program’s theory of change p.4 The evaluation cycle: three key steps p.5 Step 1: Planning an evaluation p.6 Defining the purpose and scope of an evaluation p.6 Identifying and involving stakeholders and key people p.7 What kind of evaluation do you need? p.8 Choosing your evaluation design p.9 Methods for collecting data p.10 Step 2: Implementing an evaluation p.12 Analyzing and interpreting data p.13 Step 3: Reporting and using evaluation findings p.14 Reporting results p.14 Using results p.14 Evaluating financial education programs: some additional considerations p.15 Annex p.16 Page 2 “Are we making a difference?” What is evaluation? The Money Matters Program The Money Matters Program is a group of A question often raised by people If a program is making a difference, projects that have been initiated by the implementing financial education projects and evaluation will help you understand how, to community to improve the financial capability of programs is, “How do we know we are what degree and why. adolescents (ages 13-18) through education and making a difference?” Those working  Evaluation is systematic – this means practical experiences. closely with clients and participants will likely that evaluation proceeds according to a notice changes, but it is challenging for them There are two main project streams for the plan, ideally conceived at the same time to demonstrate the nature of the change, program: as the program is being designed. whether these changes are long-term, and if  Financial “fun camps“ for 13 to 15 year these changes have an impact (or make a  Evaluation is evidence-based – olds during which youth participate in an difference). findings are based on evidence and do not after-school 2-3 hour workshop hosted by a rely on ad hoc or anecdotal observations. This guide provides an overview of local youth agency. The focus of the workshop This requires systematic procedures for evaluation – a process that allows program is on understanding the basics of budgeting, producing valid and reliable descriptions and project managers to systematically saving, and credit use while engaging with of program performance. measure change to demonstrate that they are other age-appropriate activities. making a difference. It introduces the main  Community “internships” for 16 to 18 year  Evaluation measures or makes concepts and considerations for planning and olds during which youth are paired with assessments – key to evaluation is undertaking evaluations that would be financial institutions, community agencies and deciding which aspects of the program will relevant for a wide range of financial businesses for one week to shadow and learn be assessed. Financial education education projects, programs, and initiatives. about financial aspects of these organizations, programs commonly measure such as preparing bank deposits, writing effectiveness (are we making a difference) We will be using an example throughout the cheques, tallying receipts, or developing a and efficiency (are we using resources guide to apply these concepts to a financial project budget. wisely). education program.  The three-year program has received financial  Evaluation contributes to making and in-kind support from the Community decisions – the main reason that After reading this guide, the user should: Council, government, businesses and evaluations are conducted is that the  Have a better understanding of key community organizations. Representatives of findings from the evaluation will assist in evaluation concepts and how they apply to these bodies make up the Program Steering making decisions about the program. financial education projects, programs and Committee, which provides direction and initiatives; and oversight for program operations. The  Be better prepared to design and implement program is implemented and managed by a an evaluation. community youth services agency. Page 3 In general, evaluations of projects and programs tend to focus on “Why are we doing this?” understanding inputs, activities, outputs and short term outcomes. We Understanding your program’s theory of change often better understand the impact of a specific program after some time has elapsed or when evaluating broader initiatives and policies At the outset it is helpful to map out what you expect to happen as a which involve multiple programs and projects. result of your program. One tool you can use is called a logic model. Using the example and Figure 1, it may be helpful to consider a series A logic model is a visual way of expressing the expected impact of a of questions, working backwards from where we would like to end up: program, its theory of change: doing this will cause (or contribute to)  Why are we doing this? (Impact) What will the impact be on that. It is the rationale for why your program is structured as it is. the target population as a result of this community program? Thinking about your logic model as you plan for evaluation allows you  What do we expect to achieve? (Outcomes) What changes in to revisit your objectives and consider how you will measure success. It learning or behaviour can participants expect as a result of helps to describe very clearly why you think the investments you make participating in the program? (inputs), and the particular activities that you undertake, will lead to the results you hope to achieve (sometimes called a “results chain”).  How do we think that will happen? (Inputs, Activities, Outputs) What are the program’s “deliverables”? What is Especially important is the distinction between an output (something required to accomplish these? that is under the program’s control, often measured during or shortly after the program) and an outcome (a change in a participant or the Thinking about these three questions leads naturally to considering overall environment, hopefully influenced by the program, but often what type of evaluation you will need and the methods you will choose as you try to answer the question, “How will we know if we have due to other factors as well). been successful? Figure 1: Example of a logic model EFFICIENCY EFFECTIVENESS INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACTS The financial, What the program “does”: What the program Changes that result from the Changes that result from the human and actions taken or work “produces”: the outputs; these changes are immediate outcomes, generally material resources performed through which products, goods most closely associated with considered a change in overall used by the inputs are mobilized to and services which or attributed to the project. "state." Impacts can be similar to strategic objectives. program. produce specific outputs. result from the activities. Increased proportion of Funds, staff, Recruiting youth, Participants, Increased participant young adults with some community developing materials, knowledge of the banking agencies, local workshops, workshops, system, budgeting, credit level of savings; decreased youth centers, facilitating internships, use, savings, and taxation; level of roll over of credit internship sites workshops, recruiting partnerships, increased participation card balances or defaults on credit cards by young and placing interns reports among participants in adults banking services Page 4 The evaluation cycle: three key steps Evaluation is best planned as part of program development – in the early stages of designing a new program, or a new component of an existing program. There are specific reasons for this: Figure 2: Evaluation  Planning for an evaluation requires specific decisions on what constitutes Cycle “success.” By having to define this very early on, you are more likely to design your program in a more rigorous manner to match expected outcomes. Planning an  Measuring change requires baseline information. By planning early for an Evaluation evaluation, you are able to collect information about participants before they become involved with the program and thus attribute change in learning or behaviour to the program.  Collecting evaluation data can become integrated with other program activities such as registration, assessment, and follow-up, and therefore be an efficient use Reporting and Implementing an of resources. Using Evaluation Evaluation  It is an ideal time to consult with stakeholders and build their interests into your Findings evaluation plan. As with any complex exercise, an evaluation proceeds through several stages. Figure 2 shows three main steps in the evaluation cycle. It is called a “cycle” because it is part of an ongoing process of monitoring and improvement as the program evolves. In the next few pages we will look at each of these steps more closely. Step 1: Planning an Evaluation – This stage involves Step 2: Implementing an Step 3: Reporting and Using determining the purpose and scope of the evaluation Evaluation – Implementation Evaluation Findings – The key through consultation with key people, choosing what kind involves collecting and analysing data to a successful evaluation is that of evaluation best suits your purposes, focusing on the and information according to the the results are effectively important questions the evaluation will answer and evaluation plan that was developed in communicated and used. determining what methods you will use to collect data and the program design stage. report your findings. Page 5 Step 1: Planning an evaluation Figure 3: Reasons for evaluation The evaluation process will be different for every project, program or initiative. While there are some standard approaches and methods, Reasons for Evaluations Money Matters Program there is no one right way of designing an evaluation. It is advisable to plan the evaluation at the same time as the program, but these Project and program The community youth services same steps and principles can be followed even if a program has improvement agency implementing and been running for some time. Managers want guidance and managing the program would information on how they can improve want to know how they could their programs either by making them improve the program. For Defining the purpose and scope of an evaluation more effective (making a larger example, would other types of difference) and/or how to make them activities be more effective? There are many reasons to conduct an evaluation. It is important to more efficient. be clear on why the evaluation is needed (the purpose). Figure 3 presents a range of reasons that may be given for conducting an Public accountability The Community Council, evaluation, depending on your role or perspective. Those groups who are receiving or regional government and It is also important to agree on what to include or exclude in the providing funding for the projects or financial institutions will need to evaluation, and why (the scope). programs will want the evaluation to demonstrate to the public and to demonstrate that the funding and Board Members that this  What does the program manager want to learn? resources being used are making a investment of resources has  What do funders and sponsors need to know? difference and that they are being used achieved results, or if not, why wisely. not.  Which component of the program are you planning to evaluate? A good way to decide the scope of the evaluation is to revisit the Knowledge development An evaluation of the Money program’s objectives. Objectives are what you want to accomplish Evaluations are a good source for Matters Program could in terms that are specific, measurable, achievable, reasonable and developing knowledge about various contribute to a better time-specific. Designing an evaluation involves aligning objectives types of programs, and what works or understanding of how financial with evaluation criteria, or indicators of success. Different doesn’t work. education programs work (or objectives can be evaluated differently. don’t) for adolescents, in general. The timing of the evaluation is also important to consider. How long should the program be running before it is evaluated? Sometimes Policy making Evaluating the Money Matters external factors, such as the reporting requirements of sponsors or the need to align with funding cycles, determine the timing of an At a broad level, evaluations can Program could lead to contribute to the development or developing a policy that financial evaluation. adjustment of policy. education be mandated as a key component of community economic development projects. Page 6 Brainstorming evaluation questions Identifying and involving stakeholders and key people Evaluation questions will drive your evaluation. As you engage in more detailed planning, you will narrow down the questions; they will become quite precise and suggest a particular design and Deciding on the purpose and scope If stakeholders are involved in methodology. However, at this early consultation stage, it may be of the evaluation often involves identifying the reasons for an useful to capture all of the angles of interest depending on the consulting with those individuals, evaluation and selecting the stakeholder’s perspective, and then decide what is feasible given your groups or organizations that have a questions that the evaluation will resources. significant interest in how well a try to answer, then it is more likely program functions – its that the evaluation findings will be Here are some questions that stakeholders of the Money Matters stakeholders. Stakeholders can used. Program might like an evaluation to answer: include board members, funders or  How effective was the program design? If the project was to be sponsors, administrators, staff, In the Money Matters Program, the implemented again, what should change? Stay the same? How clients and intended beneficiaries, program administrators may be could the project run more smoothly? community leaders and the public. interested in understanding the  To what extent did the program achieve what was expected? challenges in implementing the What were the primary factors contributing to success? The main program, while the funders may be Stakeholders for the Money Matters challenges encountered? The main lessons learned? more interested in questions on Program could include:  What impact did the program have on the community (e.g. new whether the resources were used  members of the Program Steering partnerships, raising the profile of financial education, increased efficiently. Committee interest in community service by local banks)?  the Community Council  How did the program affect the attitudes, behaviour, knowledge  local businesses, banks, and It is good practice to establish a and skills of participants? Were there unintended consequences agencies offering internships structure and method for and effects, (e.g. more discussion of financial matters with peers)?  program manager and staff at the How lasting was the change in participants’ learning and communicating with key people on youth services agency behaviour? an ongoing basis. Consider  youth participants and parents forming an evaluation working  To what extent do participants now have an increased knowledge group or advisory group that of the banking system? The wise use of credit? The importance of Involving stakeholders will provide includes representatives from the compound interest for debt and savings? a more comprehensive different stakeholder groups. This  To what extent has the program contributed to participants using understanding of perspectives of group should be assembled in the banking services? what the program or initiative is planning phase and meet regularly  To what extent was the program implemented as planned? Did the trying to achieve. Surprisingly, throughout the entire evaluation changes improve or detract from program results? different stakeholders often have cycle to facilitate continuing input  Was the recruitment strategy effective in attracting the target different expectations of the from key people, as well as to keep population? Do the agencies that provided the internships think primary objectives. the community informed (and the program worked? What problems were encountered and how involved). were they solved?  Is this program a good use of resources? What was the average cost per participant? Page 7 What kind of evaluation do you need? Figure 4: Evaluation questions by evaluation types There are times in the life of a program or project that you need to know certain things. What you want to know will suggest a particular kind of Evaluation Brief Description Generic Key evaluation: Types Evaluation Questions Provides answer to Is the program having the  At the outset, you may need information that establishes the need Outcome & the overall question desired effects? for the program in the first place (needs assessment); Impact “are we making a Do these effects differ by  In the middle of the program, you may want to see if it is working Evaluations difference?” group? in the way you intended (formative evaluation); and, Is the program having unintended effects  Usually at the end of a program cycle, you will want to have (positive and negative)? results that prove your program is sound (summative evaluation). These types of Are program effects It is customary to plan for such activities as a normal part of program Cost analyses relate costs attained at a reasonable development. Effectiveness & to the outcomes and cost? Other types of evaluation usually have a more specific focus. Imagine Cost Benefit impacts that are What is the average cost being achieved Analyses per outcome? putting the links of your results chain under a microscope. What do you want to know? Focuses on whether How do the services  You may need to demonstrate what impact the program is having Design the program “makes provided by the program on the problem, e.g., on the financial education of adolescents in Evaluation sense,” whether the contribute to results? the community (impact evaluation). program is likely to What are alternative ways  You may need to know how participants have benefited from the achieve results in the of delivering these services way it is designed. program (outcome evaluation). that might produce better results?  You may be required to show that it provides value for money (cost effectiveness or cost-benefit analysis). Measures how well To what extent has the  You may question whether you could achieve program objectives Implementation the program has program been in other ways (design evaluation). Evaluation been implemented, implemented as planned? and the extent to What are the main  You may want to know how well the program has been which the program challenges encountered in implemented (implementation evaluation). was implemented as implementing the You will notice in Figure 4 that by generating evaluation questions (what planned. program? do you want to know?), it becomes easier to decide what kind of evaluation is required. Page 8 Choosing your evaluation design In selecting a design, you should balance the reason for the evaluation, the resources for the program/initiative, and the intended use of the evaluation findings. The Project Steering Committee of the Money Matters Program will For the results of an evaluation to be convincing, or credible, the conduct an outcome evaluation after 30 months of the three-year evaluation has to be carefully designed. An evaluation can be more or program. They will use a pre-post design to assess knowledge and less robust depending on the design you select and the methods you practices when participants apply to either component of the program use. A robust evaluation allows you to make strong claims about your (pre) and a similar measure after they finish the program (post). program and generalize your results to other situations or individuals Finally, three months later, the program plans to follow up and determine if changes have persisted (post). beyond the program reality (e.g., youth in general). When designing the evaluation, the program managers suggested  In a true experimental design, participants are randomly that those participants who had the opportunity to participate in both assigned to a “treatment group” that participates in the program, the fun camp workshop and an internship within the 30-month period and a “control group” that does not. This is challenging to (because of their age) be analyzed separately to determine if there implement, given the realities of social programs. was an interactive effect between program components. This became an additional evaluation question of interest.  A quasi-experimental design involves comparing those who The evaluation design identified key indicators of success that aligned participated in a program with those who did not have the with program objectives, such as the number of youth opening a new opportunity to participate. Your “comparison group” would have to bank account within three months of completing the internship. This match the “treatment group” in every way so any change could be led to including questions about personal banking practices in the attributed to the program. registration process to provide baseline information against which they could measure the impact of the program.  Given what is involved in experimental designs, many programs select a non-experimental evaluation design. These are much easier to implement. For example, a pre-post design is commonly used to measure how much people have changed as a result of an intervention by assessing them before (pre) and after (post) they participate. This design can be applied to change in attitudes, learning and behaviour. Page 9 Methods for collecting data Once you have selected the overall evaluation design, you will need to Surveys: a list of questions designed to collect determine the specific methods that you want to use to collect the evidence information from participants on their knowledge and that will support your findings, or data. Some data collection methods provide perceptions of a program or service, or as a testing more depth of information, while others provide more breadth. instrument to assess knowledge and to understand common practices. Surveys offer breadth but often less  Depth – understanding the impact of a program on an individual person depth. They can be useful for large target audiences and when you want to collect quantitative data (numbers such In the Money Matters Program, you might select four participants in the internship as test scores, ratings, etc.). component and interview them to understand how they think the internship experience made a difference to their financial education. By using this method, you will learn about the activities they undertook as part of their internship, the When designing surveys for evaluations, things to keep in challenges they faced, what they view as the impacts to date. While this provides mind are: in-depth information about the factors that contribute to, or detract from, program  Keep the survey to a reasonable length (often 15-20 successes in some cases, the results cannot normally be extended to the larger group of participants. minutes is ideal).  Try to determine how many people are likely to respond – the potential response rate – even though surveys  Breadth – understanding the impact of a program on a large group of must be voluntary. If surveys have a low response rate people, but in less detail (e.g., less than 60%), they may not produce reliable data. In the Money Matters Program, you could choose to implement a knowledge test  Consider how you will administer the survey – most when participants register for the program, then again shortly after they have surveys are administered either by paper copy (mailed completed the program. These tests would provide a pre-post knowledge score for out), by phone, or on-line. Each method has different all students. This information reflects the results for all participants. However, it cost considerations and different response rates. Things does not provide the more in-depth information to understand what the challenges to consider – do the participants have telephones, what had been if, for example, the scores showed little difference. is their education level to complete a hard copy survey, do they have access to the internet? In general, evaluations benefit from being able to combine multiple lines of  Do you have access to people who can compile and evidence from different points of view (e.g., trainers, students, experts). analyse the data from the survey? Is special software or Good evaluations generally use multiple methods and multiple sources of expertise required? evidence. What follows is a description of the most common ways of collecting information to evaluate a community-based program. Page 10