Working paper 395 Monitoring and evaluation of policy influence and advocacy March 2014 Working paper 395 Monitoring and evaluation of policy influence and advocacy Josephine Tsui, Simon Hearn, and John Young Policy influence and advocacy are increasingly regarded as a means of creating sustainable policy change in international development. It is often also seen as a difficult area to monitor and evaluate. Yet there is an increasingly rich strand of innovation in options to monitor, evaluate and learn from both the successes and failures of policy influence and advocacy interventions. This paper explores current trends in monitoring and evaluating policy influence and advocacy; discusses different theories of how policy influence happens; and presents a number of options to monitor and evaluate different aspects of advocacy interventions. Case studies describe how some organisations have used these options in practice to understand their impact and improve their advocacy strategies. Shaping policy for development odi.org Acknowledgements The authors would like to thank all those who contributed to the working paper. First among them are the directors and staff of the Bill & Melinda Gates Foundation who so willingly gave us their time including Ritu Shroff, Amy Pennington, Nushina Mir, Jodi Nelson and Oliver Babson. We couldn’t have done this work without the staff at TCC Group. The authors drew heavily on the research and expertise of the members of the Research and Policy in Development (RAPID) team at Overseas Development Institute (ODI). We are grateful for the helpful comments provided by peer reviewers Jared Raynor, Ritu Shroff and Amy Pennington. Many thanks to the support staff at ODI and the Bill & Melinda Gates Foundation who helped with administration, logistics, lay-out, formatting and editing. We are grateful to Jessica Sinclair Taylor and Anna Brown for lay-out and editing and Jared Raynor for external review. This working paper is based on research funded by the Bill & Melinda Gates Foundation. The findings and conclusions contained within are those of the authors and do not necessarily reflect positions or policies of the Bill & Melinda Gates Foundation. Cover photograph: Community youth air their views at a meeting in Matsekope, Ghana. Nyani Quarmyne/PDA Ghana. Title page photo: Workshop in Cambodia. Arnaldo Pellini/ODI. Table of contents Acknowledgements ii Abbreviations iii Introduction 1 Before you start 1 How this paper is organised 4 1 Current trends in monitoring and evaluating policy influence & advocacy 5 1.1 Policy influence and advocacy and M&E 5 1.2 Current issues and debates 8 1.3 Conclusion 11 2 Approaches and frameworks for understanding policy influence 1 2.1 Understanding policy and political economy context: policy development and formulation frameworks 3 2.2 Frameworks to guide intervention planning: understanding tactics 4 2.3 Frameworks to understand levels of influence 6 2.4 Conclusion 7 3 Methods and tools for monitoring and evaluating advocacy 8 3.1 Strategy and direction 10 3.2 Management and outputs 11 3.3 Outcomes and impact 12 3.4 Understanding causes 13 3.5 Conclusion 15 4 Case studies 16 4.1 Evaluating value for money of DFID’s influence in the health sector 17 4.2 Publish What You Pay Coalition self-assessment 18 4.3 Creating an M&E dashboard to monitor online influence 19 4.4 Supporting international climate negotiators 20 4.5 The stealth campaign overthrew the juvenile death penalty 21 4.6 Evaluating the European Development Cooperation Strengthening Programme’s influence 22 5 Conclusions 24 Appendix A: Frameworks for understanding policy influence 25 A1 Frameworks to understand context 25 A2 Frameworks to guide intervention planning 31 A3 Frameworks to understand levels of influence 38 Appendix B: Options for monitoring and evaluating advocacy 41 B1 Options for evaluating strategy and direction 41 B2 Options for monitoring management and outputs 48 B3 Options for monitoring and evaluating outcomes and impact 53 Monitoring and evaluation of policy influence and advocacy i B4 Options for understanding causes 58 References 64 Figures Figure 1: The three spheres of control and their relationship to policy influence 1 Figure A1: The policy circle (Hardee et al., 2004) 27 Figure A2: The Context, Evidence, Links framework 30 Figure A3: Supporting advocacy strategy framework 31 Figure A4: K* framework 34 Figure A5: Rogers theory of diffusion (Robinson, 2009) 37 Figure A6: Outcomes to advocacy strategies 40 Figure A7: Example problem tree 42 Figure A8: Stakeholder interest and influence matrix 43 Figure A9: Example of an SNA map (Davies, 2009) 44 Figure A10: Stakeholder analysis diagram (ODI) 56 56 Figure A11: Description of the four different tests used for process tracing (Collier, 2011) 59 Figure A12: A completed ROA map 62 Figure A13: Example of a truth table, correlating all the causal conditions with respect to IMF conditionality and protests. 63 Tables Table 1: Brief description of policy development and formation frameworks 3 Table 2: Brief description of frameworks to guide intervention planning 5 Table 3: Frameworks to gauge the levels of influence interventions have on policy change 6 Table 4: List of methods that focus on strategy and direction 10 Table 5: List of methods that focus on management and outputs 11 Table 6: List of methods that focus on outcomes and impact 12 Table 7: List of methods that focus on understanding causes 14 Table 8: Five levels of competency 18 Table 9: Assessment benchmark against the logic framework 19 Table 10: An example of CDKN’s progress markers and how they have been colour-coded to show evidence of change. 20 Table 11: Example of outcome mapping method to determine links of EDCSP’s outputs to events and impact 23 Table A1: Sample logframe table 43 Table A2: Coalition Capacity Checklist 48 Table A3: An example of World Vision’s Influence and Engagement tool using a scalar model 50 Table A4: Example of MDCA decision scoring 52 Table A5: Examples of stakeholders 56 Monitoring and evaluation of policy influence and advocacy ii Abbreviations CDKN Climate and Development Knowledge Network EDCSP European Development Cooperation Strengthening Programme GPA Global Policy Advocacy M&E Monitoring and evaluation MCDA Multi-criteria decision analysis ODI Overseas Development Institute QCA Qualitative Comparative Analysis RAPID Research and Policy in Development programme SNA Social network analysis Monitoring and evaluation of policy influence and advocacy iii Introduction International development organisations are increasingly turning to policy influence and advocacy work as a means of realising sustainable, transformative change. As advocacy work increases in prominence, organisations are also looking for ways to monitor and evaluate the changes they are hoping to produce. This ODI Working Paper presents a comprehensive review of the theory and practice of monitoring and evaluating advocacy and policy influence. It begins with a summary of the key debates in the field to define what makes the monitoring and evaluation (M&E) of advocacy distinct from other kinds of M&E. It presents different frameworks that are used to understand the policy context, advocacy interventions and the influences that these have on policy. Finally it presents a number of practical tools, methods and approaches that are commonly used for the M&E of advocacy. Some of these approaches are explored in practice through six case studies. This working paper was commissioned by the Bill and Melinda Gates Foundation to inform their measurement and evaluation work around global policy and advocacy. The Foundation’s Global Policy and Advocacy (GPA) division supports a wide range of foundation interests (e.g., Agriculture, Vaccine Delivery, Family Planning, Polio etc.) primarily by creating effective policy and advocacy strategies, leading foundation engagement on a number of important cross-cutting issues, such as aid reform, and coordinating engagement strategies in the most important donor and developing country geographies. The paper builds on preparatory policy briefs produced for the Foundation’s ‘Monitoring and Evaluation of Policy and Advocacy Convening’ on the 10th and 11th of December, 2013. Before you start Before diving into the tools and options for understanding policy influence and the M&E of advocacy, it is useful to think through the following five stages1 of planning an M&E system for an advocacy intervention. 1. How do you FRAME the M&E: Why are you monitoring and evaluating, for whom, and on what basis? For M&E findings to be useful, the process has to be driven by an explicit and commonly understood purpose. Be clear in advance who are the users of the M&E data, analysis and findings and how they will use them. Chapter 1 describes how thinking about the purposes of M&E has evolved, from a sometimes ‘tick box’ approach to donor accountability, to an approach which puts equal emphasis on learning and accepts the role of failure and mis-steps as a valid learning tool. Other 1 Based on the BetterEvaluation Rainbow Framework: http://betterevaluation.org/plan Monitoring and evaluation of policy influence and advocacy 1 common purposes that go beyond the accountability and learning purposes include increasing knowledge, building capacity, trust and allegiances. 2. How do you DEFINE what you are monitoring and evaluating: What is the intervention or set of interventions and how is it understood to work? Whether you are planning or commissioning an evaluation or designing an M&E system, one of the first tasks is to define what is to be evaluated and how it is understood to work. For advocacy, this means defining the strategies, the activities, the outputs and the intended outcomes – all of which need to be monitored and evaluated to some extent. Chapter 2 presents many approaches to understanding and defining how your advocacy intervention will achieve its aims. The policy and political economy context frameworks in section 2.1 will form the basis of your understanding about how policy develops and changes in your context, who is involved and how. The advocacy intervention frameworks in section 2.2 are the menu from which you decide the appropriate mix of tactics for engaging in this context to influence the right people in the right way and to achieve your ultimate aim. The influence frameworks in section 2.3 can be used as models to define the expected short-term and long-term outcomes. 3. How do you DESCRIBE what happened: Activities, outputs, outcomes, context? There are many options for monitoring an advocacy intervention’s strategy, activities, outputs and intended outcomes, some of which are presented in Chapter 3: theory of change, logical framework, outcomes hierarchy, impact pathway, programme theory and logic model. They all share the same broad purpose: to develop a shared, explicit understanding of how things are understood to change, how the intervention will engage to support certain changes and/or inhibit others, what causal steps are involved, what assumptions are being made about this and what rationale we have for making these causal claims. Developing some kind of theory of change not only helps in planning the intervention strategically but it will also help prioritise evaluation efforts. The most basic M&E system is one that can tell you at any given time what has been done by the programme, where, when, with whom, what the aim was and what actually happened. More sophisticated M&E systems will also be able to tell you the effects of those actions – how did key actors react? what kinds of changes were observed? what kind of secondary or knock-on effects have been observed – positive or negative? In addition to this, a good M&E system will also provide information on the context within which these activities were conducted or change was observed. This kind of descriptive data is essential for the majority of purposes of M&E and should be the minimum standard for most M&E systems. Section 3.2 in Chapter 3 presents a number of techniques that are commonly used for collecting and analysing descriptive data, for example, journals, logs, dashboards, bellwether interviews, most significant change and rubrics. 4. How do you understand what CAUSED the observed outcomes and why they happened? Sometimes you want to do more with M&E than just answer questions about what happened: you want to go further to understand what caused the observed outcomes and impacts, and to what extent the advocacy intervention contributed to the Monitoring and evaluation of policy influence and advocacy 2 outcomes. This kind of analysis is usually reserved for evaluations, whether internal or external, but there is a rudimentary level of informal analysis of causes that goes on throughout the monitoring process – as team members, managers and partners meet and discuss progress and make sense of the data being collected. The methods and approaches presented in section 3.4 of Chapter 3 are the most commonly used for understanding causes. In order to understand causes in advocacy, it is recommended to spend time at the start thinking through a theory of change – as discussed in point 2 above. Having a theory to compare data against means it is easier to gather the right kind of data to establish causal relationships, thereby increasing efficiency. 5. How do you ASSESS the overall success? Evaluations are ultimately about making an evidence-based judgement about the merit, worth or performance of a programme or intervention. Synthesising the evidence from the kinds of analyses described in points 3 and 4 above is a crucial part of the evaluation. An evaluator or programme officer may have to weigh up successes in some parts of the advocacy initiative and failures in others. They may be confronted with partial successes where the desired outcome has only partly been observed; or it was observed but there was insufficient evidence to demonstrate contribution from the intervention; or maybe it took longer than expected with a lot more resources used to produce the outcome. In each of these scenarios, careful assessment is required to make an overall judgement. The ambitions and criteria used for this should be considered part of the development of an M&E system because different synthesis approaches will have different requirements. The case studies in Chapter 4 present some examples of how evaluation data from different projects was synthesised in order to draw conclusions about the success of the intervention. Monitoring and evaluation of policy influence and advocacy 3 How this paper is organised Chapter 1 provides a general overview of current trends in monitoring and evaluating advocacy and policy influence, including current debates on the purpose of M&E. The chapter also covers the influence of complexity theory on M&E of advocacy and the tension between attribution and contribution: how far a given change can be attributed to one intervention, or whether it is better to speak only of an intervention’s contribution to a change. Chapter 2 provides a set of frameworks for understanding the theory of policy influence, covering:  frameworks to understand the context of policy influence  frameworks to understand tactics and guide intervention planning  frameworks to understand the process of influence: how change occurs. Chapter 3 covers options and considerations for monitoring and evaluating advocacy. The chapter presents options for the M&E of all four areas:  evaluating the strength of an intervention’s theory of change or strategy  monitoring management and outputs to find out how well an organisation is carrying out its plan  tracking outcomes and impact: how much change has occurred  understanding causes: how the intervention might have caused or helped to cause the observed changes and the other factors that were influential. To help the reader understand how some of the M&E options presented could be implemented in practice, Chapter 4 describes six case studies of how organisations have evaluated different kinds of advocacy and policy influence programmes. Monitoring and evaluation of policy influence and advocacy 4