FINAL REPORT Evaluation Design Report for the Benin Power Compact's Electricity Generation Project and Electricity Distribution Project March 7, 2018 Christopher Ksoll Kristine Bos Arif Mamun Anthony Harris Sarah Hughes Submitted to: Millennium Challenge Corporation 1099 14th Street, NW Suite 700 Washington, DC 20005 Project Monitor: Hamissou Samari Contract Number: MCC-16-CON-0058 Submitted by: Mathematica Policy Research 1100 1st Street, NE, 12th Floor Washington, DC 20002-4221 Telephone: (202) 484-9220 Facsimile: (202) 863-1763 Project Director: Sarah Hughes Reference Number: 50339.01.200.031.000 This page has been left blank for double-sided copying. BENIN ENERGY II: EVALUATION DESIGN REPORT MATHEMATICA POLICY RESEARCH CONTENTS ACRONYMS ............................................................................................................................................... VII I. INTRODUCTION ........................................................................................................................................ 1 II. OVERVIEW OF THE COMPACT, GENERATION ACTIVITY, AND DISTRIBUTION ACTIVITY ......................................................................................................................................... 5 A. Overview of the Benin Power Compact ..................................................................................... 5 B. Overview of the theory of change .............................................................................................. 9 C. Economic rate of return and beneficiary analysis.................................................................... 10 III. LITERATURE REVIEW.................................................................................................................. 13 A. Grid-level outcomes ................................................................................................................. 15 1. Increased supply of electricity ........................................................................................... 15 2. Improved electricity quality and reliability.......................................................................... 15 B. Business outcomes ................................................................................................................. 16 1. Longer hours of operation for businesses ........................................................................ 16 2. Lower reliance on costlier energy sources........................................................................ 16 3. Reduced degradation of equipment due to unstable electricity ........................................ 17 4. Reduced losses of products and perishable goods .......................................................... 17 5. Increased productivity for businesses ............................................................................... 17 C. Household outcomes ............................................................................................................... 17 1. Lower reliance on costlier energy sources........................................................................ 17 2. Improved education outcomes and increased productivity for households ...................... 18 D. Gaps in the literature and contribution of the Benin II Energy evaluation ............................... 19 IV. EVALUATION DESIGN.................................................................................................................. 21 A. Evaluation questions ............................................................................................................... 21 B. Overview of the evaluation strategy ........................................................................................ 22 V. QUANTITATIVE ANALYSIS OF GRID-LEVEL OUTCOMES ........................................................ 27 A. Methods ................................................................................................................................... 27 1. Interrupted time-series (ITS) approach. ............................................................................ 27 2. Pre-post and descriptive analysis ..................................................................................... 29 B. Outcomes and their anticipated time frame for realization ...................................................... 30 1. Short-term outcomes......................................................................................................... 30 2. Medium- and longer-term outcomes ................................................................................. 32 C. Analysis plan and statistical power .......................................................................................... 32 FINAL iii BENIN ENERGY II: EVALUATION DESIGN REPORT MATHEMATICA POLICY RESEARCH 1. ITS approach ..................................................................................................................... 32 2. Pre-post analysis ............................................................................................................... 35 D. Data sources............................................................................................................................ 36 1. Placement and measurement on a simple network ............................................................ 36 2. Additional considerations for grid monitoring .................................................................... 38 3. Data collection ................................................................................................................... 40 E. Challenges ............................................................................................................................... 41 VI. QUANTITATIVE ANALYSIS OF END-USER OUTCOMES .......................................................... 43 A. Methods ................................................................................................................................... 43 1. ITS approach ..................................................................................................................... 43 2. Pre-post approach with longitudinal data .......................................................................... 43 B. Outcomes and their anticipated time frame for realization ...................................................... 44 1. Short-term outcomes......................................................................................................... 44 2. Longer-term outcomes ...................................................................................................... 45 C. Analysis plan and statistical power .......................................................................................... 46 1. ITS analysis ....................................................................................................................... 46 2. Pre-post analysis with longitudinal data ............................................................................ 47 D. Data sources............................................................................................................................ 48 1. Sampling ........................................................................................................................... 50 2. Data collection approach .................................................................................................. 51 E. Challenges ............................................................................................................................... 52 VII. QUALITATIVE ANALYSIS .................................................................................................................... 55 A. Methodology ............................................................................................................................ 55 1. Implementation analysis ................................................................................................... 55 2. Qualitative assessment ..................................................................................................... 56 B. Time frame of exposure ........................................................................................................... 56 C. Study sample ........................................................................................................................... 57 D. Approach to data collection ..................................................................................................... 59 E. Analysis plan............................................................................................................................ 62 F. Challenges ............................................................................................................................... 63 VIII. ADMINISTRATIVE ISSUES ........................................................................................................... 65 A. Summary of IRB requirements and clearances....................................................................... 65 B. Data access, privacy, and file preparation .............................................................................. 65 FINAL iv BENIN ENERGY II: EVALUATION DESIGN REPORT MATHEMATICA POLICY RESEARCH C. Dissemination plan .................................................................................................................. 66 D. Evaluation team: Roles and responsibilities ............................................................................ 66 E. Evaluation timeline and reporting schedule ............................................................................. 66 REFERENCES ............................................................................................................................................ 69 APPENDIX A OPTIONAL QUANTITATIVE ANALYSIS: IMPACT EVALUATION OF STRATEGIES TO ENCOURAGE NEW CONNECTIONS ................................................. 79 APPENDIX B DETAILED SUMMARY OF PLANNED ACTIVITIES AND INVESTMENTS ....................... 95 APPENDIX C NETWORK SCHEMATICS ............................................................................................... 101 APPENDIX D STAKEHOLDER COMMENTS AND MATHEMATICA RESPONSES .............................. 105 FINAL v BENIN ENERGY II: EVALUATION DESIGN REPORT MATHEMATICA POLICY RESEARCH TABLES Table II.1. Benin Energy Compact: Activities and investments .................................................................... 6 Table IV.1. Evaluation questions and links to program logic ...................................................................... 21 Table IV.2. Evaluation design overview ...................................................................................................... 24 Table V.1. Minimum detectable impacts for grid-level outcomes under the ITS design ............................. 35 Table V.2. Grid monitors: Possible placement locations and measures .................................................... 38 Table VI.1. Proposed short-term end-user outcomes for ITS analysis ....................................................... 44 Table VI.2. Minimum detectable impacts for ITS analysis .......................................................................... 47 Table VI.3. Quantitative sampling and data collection ................................................................................ 49 Table VII.1. Key areas of focus for qualitative data collection, by respondent ........................................... 60 Table A.1. Minimum detectable impacts for new connections .................................................................... 88 Table A.2. Optional connection study sampling and data collection .......................................................... 90 Table B.1. Summary of Electricity Generation and Distribution Project activities and investments ........... 97 Table D.1: Stakeholder Comments on the Design Report and Mathematica Responses ........................ 107 FIGURES Figure II.1. Map of project regions ................................................................................................................ 8 Figure II.2. Theory of change for the Electricity Generation and Distribution Projects ................................. 9 Figure V.1. Interrupted time-series approach ............................................................................................. 28 Figure V.2. A simplified electricity network ................................................................................................. 37 Figure VIII.1. Evaluation timeline and reporting schedule .......................................................................... 67 Figure C.1. Parakou network schematic ................................................................................................... 103 Figure C.2. Natitingou network schematic ................................................................................................ 104 FINAL vi BENIN ENERGY II: EVALUATION DESIGN REPORT MATHEMATICA POLICY RESEARCH ACRONYMS AMI Advanced Meter Infrastructure AMR Automatic Meter Reading BTI Business and Technology International CAPI Computer-Assisted Personal Interviewing CEB Communauté Electrique du Benin ERR Economic Rate of Return FGD Focus Group Discussions GDP Gross Domestic Product GoB Government of Benin GPS Geospatial Positioning System GSI Gender and Social Inclusion GW Gigawatts HV High Voltage IEA International Energy Agency IEG (World Bank) Independent Evaluation Group IFC International Finance Corporation IGA Income-Generating Activity INSAE Institut Nationale de la Statistique et de l'Analyse Economique IPP Independent Power Producer IRB Institutional Review Board ITS Interrupted Time-Series ITT Intent-to-Treat kV Kilo Volt kWh Kilowatt-Hour M&E Monitoring and Evaluation MCA-Benin Millennium Challenge Account-Benin II MCC Millennium Challenge Corporation MDI Minimum Detectable Impact MV Medium Voltage MW Megawatt MWh Megawatt Hour NA Not Applicable NDCC National Dispatch Control Center PV Photovoltaic RCT Randomized Controlled Trial RD Regression Discontinuity RFP Request for Proposals RQ Research Question SAIDI System Average Interruption Duration Index SAIFI System Average Interruption Frequency Index SBEE Société Béninoise d’Energie Electrique SCADA Supervisory Control and Data Acquisition TOT Treatment on the Treated UN United Nations FINAL vii This page has been left blank for double-sided copying. BENIN ENERGY II: EVALUATION DESIGN REPORT MATHEMATICA POLICY RESEARCH I. INTRODUCTION Similar to many countries across sub-Saharan Africa, Benin is severely lacking in electrical power, with critical deficiencies in energy access, installed capacity, and overall consumption. Benin ranks among the lowest in total net electricity generation globally (at 192 out of 198 countries) and depends heavily on imported fuel and electricity, importing 85 percent of its petroleum products from Nigeria and the vast majority of its electricity from Nigeria, Ghana, and Cote d’Ivoire (World Bank 2016). The country also relies heavily on nonrenewable sources of energy, such as firewood, charcoal, and petroleum products. Electricity access is low and uneven: Benin has a household electrification rate of only 28.2 percent nationally and 4.5 percent in rural areas (Climate Investment Funds 2015; International Energy Agency 2015; U.S. Energy Information Administration 2014). Due to its reliance on imports and inability to develop adequate electricity infrastructure, Benin has until recently suffered from a widespread scarcity of electricity. This situation changed with the government’s large-scale installation of rental electricity generation capacity in the form of diesel-powered generators, but at a substantially higher cost. Benin’s electricity supply challenges stemming from both frail electricity infrastructure and the weak financial condition of the electric utility pose a significant barrier to economic progress. Due to political pressures to keep electricity retail prices low, the cost of supplying electricity has consistently exceeded the tariff paid by consumers. The low quality of the supply of electricity in Benin, characterized by voltage fluctuations and outages, also limits usage and further reduces revenue for Benin’s electricity distribution company, Société Béninoise d’Energie Electrique (SBEE) (Louw et al. 2008). This situation has led to a history of deficits and restricted maintenance for SBEE, which in turn resulted in further deterioration of electricity infrastructure and equipment, an inability to prevent electricity shortages, and inadequate resources to expand the grid. Although SBEE’s financial situation has improved modestly in recent years, both the costs of service and deficit have been increasing (The World Bank 2017c). Overall, this poor electricity infrastructure hinders economic growth in Benin, as businesses, public institutions, and households are constrained by power outages and lack of access to the grid. To address some of these challenges, the Millennium Challenge Corporation (MCC) has partnered with the Government of Benin (GoB) to implement the Benin Power Compact (also known as the Benin II Energy Compact) from 2017 to 2022. The compact aims to modernize Benin’s electricity network, strengthen SBEE, extend access to electricity, and improve the quality and reliability of the electricity system. MCC has contracted with Mathematica Policy Research (Mathematica) to conduct an evaluation of two of the four compact activities: (1) the Electricity Generation Project, which aims to increase domestic generation capacity through new solar plants and possibly a thermal power plant, and the rehabilitation of an existing hydroelectric plant; and (2) the Electricity Distribution Project. The latter aims to strengthen the grid through upgrades; repairs; installing new switchgears, lines, and connections; and constructing a new national electricity distribution control center, as well as connecting 10,000 new households to the grid in Cotonou, Benin’s largest city, and possibly other project areas.1 1 The other two compact projects are the Policy Reform and Institutional Strengthening Project, which aims to improve the governance and management of the electricity sector, and the Off-Grid Electricity Access Project, which will finance off-grid renewable energy systems. FINAL 1 BENIN ENERGY II: EVALUATION DESIGN REPORT MATHEMATICA POLICY RESEARCH The evaluation of these two projects involves a comprehensive mixed-methods approach, which will seek to measure the impacts of and understand the changes related to the Electricity Generation and Distribution Projects. Where relevant, we will disaggregate the evaluation by key demographic dimensions to understand the programs’ effects on gender and social inclusion (GSI). We will use quantitative data to address several research questions through impact analyses (using an interrupted time-series approach and conducting an optional randomized control trial) and a performance evaluation (using a pre-post analysis). We also will conduct a qualitative performance evaluation to answer the remaining research questions related to implementation and outcomes. Quantitative impact and performance evaluation. We will use an interrupted time- series (ITS) design to estimate impacts on grid and end-user outcomes. The ITS of grid-level outcomes will leverage high-frequency data from data-logging monitors and smart meters installed on the electricity network, combined with information about project rollout, to evaluate changes in key measures of electricity reliability and quality. The continuous collection of grid- monitoring data and data from smart meters will allow us to assess impacts on grid-level outcomes, such as voltage stability and power outages. In addition, Mathematica will use the information on grid-level outcomes in combination with high-frequency phone survey data to conduct an ITS analysis of the impact of compact improvements on the short-term outcomes of households and small, medium, and large businesses. These methods will provide rigorous evidence of the impact of generation and distribution improvements on grid-level and end-user outcomes. Mathematica also proposes an optional impact evaluation to leverage the grid expansion in Cotonou and other project areas to study strategies encouraging urban households to connect to the grid. In the proposed approaches, we would use a randomized control trial or a regression discontinuity design to rigorously estimate the impacts of different pricing strategies on connecting urban households to the grid and the subsequent impact of connecting on other outcomes. To understand medium- and longer-term outcomes, we will complement the impact analyses with a quantitative performance evaluation (a pre-post analysis) of how grid- level, household, and business outcomes change over time. Qualitative performance evaluation. Mathematica will also answer research questions related to project implementation and sustainability using (1) an implementation analysis, which will explore the compact’s processes and activities through document review and interviews; and (2) a qualitative assessment, which will use interviews and focus group discussions to understand long-term outcomes for households, businesses, and public institutions. These evaluations will provide additional information to contextualize and interpret the quantitative findings, and help answer research questions that do not lend themselves to impact evaluations. In the chapters that follow, we provide context for the evaluation and describe its planned design in further detail. In Chapter II, we present the program logic and describe the activities of the Electricity Generation and Distribution Projects, and the economic rate of return (ERR). In Chapter III, we review the existing literature on improving electricity supply, quality, and reliability, and the impact on households and businesses. In Chapter IV, we outline the research questions that the evaluation seeks to answer and provide an overview of the quantitative and qualitative evaluation designs and data sources that will enable us to answer these questions. Next, we describe the quantitative evaluation designs for estimating the impacts of electricity improvements on grid outcomes (Chapter V) and end-user outcomes (Chapter VI). In Chapter VII, we describe design and data sources for the qualitative performance evaluation. We FINAL 2