' -! '^''MZNT INSTITUT« 1:9 © di Library J g Overseas Development Institute FOR REFERENCE ONLY JUDGING SUCCESS EVALUATING NGO APPROACHES TO ALLEVIABNG POVERTY IN DEVELOPING COUNTRIES Resiilts ol OiH rofeaeeh pesetited in prettnUiiary torm fof diti^ssiofi M eritical oomnient ODI Working Papers available at May 1991 13: Small Farmer Services in India: A Study of Two Blocks in Orissa Stale John Howell, 1984,- £5.00, ISBN 0 85003 090 0 15: Macroeconomic Stabilisation, Income Distribution and Poyerty: a Preliminary Survey Tony Addison and Uonel Demery. 1985. £6.00, ISBN 0 85003 094 3 16: Land Tenure Issues in Irrigation Planning Design and Management in Sub-Saharan Africa Uary T\ffen, 1985, £3.00, ISBN 0 85003 096 X 17: Balance of Payments Adjustment and Developing Countries Tony KiUick. 1985, £3.00, ISBN 0 85003 097 8 18: The Third Worid's Import Needs and Potential: A European PerspecUve, Igor Karmiloff, 1986. £6.00. ISBN 0 85003 101 X 20: Colombia, 1970-85. Management and Consequences of Two Large External Shocks Juan Jose Ediavarria, 1987. £5.00. ISBN 0 85003 109 5 21: Trade and Fuuuidiv Strategies: A Case Study of MaUysia MohamedArif, 1987, £5.00, ISBN 0 85003 108 7 22: Trade and Financing Strategies for Thailand in the 198Qs. Supote auauuuMtathim, 1987. £5.00. ISBN 0 85003 110 9 24: Industrialisation in Sub-Saharan Africa: Country case study: Cameroon Igor Karmiloff, 1988. £3.00. ISBN 0 85003 112 5 25: Industrialisation in Sub-Saharan Africa: Country case study: Zimbabwe Roger RiddeU, 1988. £3.00. ISBN 0 85003 113 3 26: Industrialisation in Sub-Saharan Africa: Country case study: Zambia Igor Karmiloff. 1988, £3.00, ISBN 0 85003 114 1 27: European Community Trade Barriers to Tropical Agricultural Products Michael Davenport. 24, 1988. £4.00. ISBN 0 85003 117 6 28: Trade and Financing Strategies for the New NICS: the Peru Case Study Jurgen SdutldtL, 1988, £3.00, ISBN 0 85003 118 4 29: The Control of Money Supply in Developing Countries: China, 1949-1988 Anita Sanlorum, 1989, £3.00. ISBN 0 85003 122 2 30: Monetary Policy Effectiveness in COte d'lvoire Christopher E. Lane, 1990. £3.00, ISBN 0 85003 125 7 31: Economic Development and the Adaptive Economy Tony KiUick. 1990, £3.50, ISBN 0 85003 126 5 32: Principles of policy for the Adaptive Economy Tony KiUick, 1990, £3.50, ISBN 0 85003 127 3 33: Exdiange Rates and Structural Adjustment Tony KiUidc, 1990. £3.50. ISBN 0 85003 128 1 34: Mari(ets and Governments in Agricultural and Industrial Adjustment Tony Killick, 1990. £3.50, ISBN 0 85003 129 X 35: Financial Sector PoUdes in the Adaptive Economy Tony KiUick, 1990. £3.50. ISBN 0 85003 131 1 36: Problems and limitations of atUustment policies Tony KiUick, 1990, £3.50, ISBN 0 85003 132 X 37: Judging Success: Evaluating NGO Income-Generating Projects Roger RiddeU, 1990. £3.50. ISBN 0 850D3 133 8 38: ACP Export DiveniTication: Non-Traditional Exports from Zimbabwe Roger RiddeU, 1990. £3.50. ISBN 0 85003 134 6 39: Monetary Policy in Kenya, 1967-88 Tony KiUick and F. M. Mwega, 1990, £3.50, ISBN 0 85003 135 4 40: ACP Export Diversification: Jamaica, Kenya and Ethiopia Christopher Stevens. 1990, £3.50, ISBN 0 85003 136 2 41: ACP Export DiveniHcation: The Case of Mauritius Matthew McQueen, 1990, £3.50, ISBN 0 85003 137 0 42: An Econometric Study of Selected Monetary Policy Issues in Kenya F. M. Mwega, 1990, £3.50, ISBN 0 85003 142 7 43: Differences in Economic Performance between Franc Zone and other sub-Saharan African Countries Christopher E. Lane and Sheila Page, 1991. £3.50, ISBN 0 85003 148 6 44: Monetary Policy Effectiveness in Indonesia, 1974-1900 Christopher E. Lane. David C. Cole and Betty F. Slade, 1991, £3.50, ISBN 0 85003 150 8 WORKING PAPER 37 JUDGING SUCCESS EVALUATING NGO APPROACHES TO ALLEVIATING POVERTY IN DEVELOPING COUNTRIES Roger Riddell May 1990 ISBN 0-85003-133-8 OVERSEAS DEVELOPMENT INSTITUTE Regent's College Inner Circle, Regent's Park LONDON NW1 4NS Acknowledgements ODI Working Papers present in preliminary form work resulting from research undertaken under the auspices of the Institute. The views expressed are those of the authors and do not necessarily reflect the views of the ODI. Coroments are welcomed and should be sent directly to the author. Preface This Working Papier is the first (preUminary) outcome of a major ODI research project evaluating the effectiveness of projects of non-government organisations (NGOs) in raising the economic status of the poor in four countries: Bangladesh, India, Uganda and Zimbabwe. It attempts to provide a framework for detailed evaluations of some 16 different projects funded, and some cases run, by British NGOs in these four countries. The studies are being undertaken during 1990. In drawing up this Paper on evaluation, the author has tried to outline an approach which is both loose enough to allow country and project sjjecific characteristics to be preserved, but detailed enough to allow some sort of cross-country conclusions to be made. The details will thus differ both from country to country and from project to project. The approach is based on the assumption that it is essential to incorporate a range of non-economic elements, relevant to the projects and the communities participating in these projects, in the overall evaluation. It is also drawn up with the knowledge that only rarely do NGO projects funded by British NGOs have base-line data of a quality sufficient for rigorous quantitative comparisons of changes in income, consumption or assets to be made. The Paper is Hkely to be revised and re-ordered as a result of the experience gained in the field evaluations. Thereafter it is hoped that if the evaluation process used here has proved useful to NGOs, it could provide at least a framework for a more general approach to similar evaluations. However even at this stage, comments would be most welcome and should be addressed directly to the author. In its present form the Paper has benefited from detailed comments from a number of people including John de Coninck, Adrian Hewitt, John Howell, Ann Muir, Paul Spray, Mark Robinson and Sarah White. The overall NGO project is being financed from a number of sources including the Evaluation Department and the Economic and Social Committee for Research (ESCOR) of the Overseas Development Administration (ODA), the Leverhulme Trust, Barclays Bank, and five of the larger British NGOs working in the development field. CONTENTS Page L INTRODUCnON 1 DL DOUBTS, CONCERNS AND FEARS 4 II. 1 Problems and Gaps in Evaluation of Official Aid 4 II. 2 Evaluation of NGO Projects and Prograimnes and Grounds for Caution 5 II. 3 Reasons for Change 9 11. 4 The Way Ahead? 11 in. TOWARDS AN APPROACH TO EVALUATING NGO POVERTY ALLEVIATING PROJECTS AND PROGRAMMES 13 IV. JUDGING SUCCESS OF POVERTY ALLEVLVTING PROJECTS OF NGOS 16 rV. 1 Introduction 16 rV. 2 Preliminary Project Familiarisation 19 IV. 3 Initial Visit to Project and Environs 19 IV. 4 Drawing Initial Conclusions and Data-Gap Identification 20 IV.5 Gathering Further Information 23 IV. 6 Refining Conclusions and Writing Up 26 V. CHECK-LIST OF QUESTIONS AND APPROACHES FOR DISCRETE NGO PROJECT EVALUATIONS 28 V. 1 Introduction 28 V. 2 Gender Issues 29 V. 3 Overall Assessment of a Project 30 V. 4 Evaluating Benefits 31 V. 5 Project Costs 36 V. 6 Marrying Costs and Benefits 40 V. 7 RepUcation and Sustainability 42 VL THE COUNTRY ASSESSMENT 46 VI. 1 General Objectives 47 VI. 2 Check-List of Specific Questions to Raise or Data to Look For 49 VI. 3 Conclusions 53 BIBUOGRAPHY 54 L INTRODUCTION Non-governmental Organisations (NGOs) which devote their attention to developing countries - Oxfam, Christian Aid, ActionAid, Cafod, the Save the Children Fund and scores of smaller NGOs - are now household names in Britain, as are their counterparts in other industrial countries. As their name suggests, over 70% of NGO income comes from the voluntary contributions of individuals, and this support is based firmly on the belief that these organisations have a positive, and many would argue a significant, impact in helping the millions of poor people inhabiting the globe, especially those Ihdng in Africa and Asia. Not surprisingly, therefore, the raison d'itre of these NGOs is, at its broadest, to work for the alleviation and eventual elimination of poverty and distress in developing countries. One of the oldest NGOs which is today concerned with developing country problems is the Save the Children Fund: it was launched in London in May 1919. The biggest British NGO, in terms of income, is Oxfam, which was started in 1942. As its original name - the Oxford Committee for Famine Relief - suggests, it was established to assist when emergencies struck.' Providing help to the victims of emergencies - in famine, earthquake, flood or hurricane - remains an important part of Oxfam's activities today as with the work of most of the major NQOs, but it is by no means their only concern. A clear distinction is now commonly made between their relief and emergency work on the one hand and their development work on the other, while it is also widely agreed that the best way to alleviate and, hopefully, eliminate poverty is to promote the long-term development of the poor - one of the beneficial effects of which is to reduce vulnerability to emergencies and disasters. As a result, to the extent that the pressures to respond to the immediate problems of emergencies permit, most NGOs devote as many resources as they can to promoting long term development.^ Working to alleviate poverty through the promotion of long term development is itself a far from simple task: NGOs and development specialists alike acknowledge that a range of different factors either inhibit or assist the development process. A useful distinction is frequently made, however, between policies and institutions within countries which have a bearing (positive or negative) on development and poverty alleviation, and policies and institutions located outside particular countries. External factors, principally those concerning aid and trade issues, are themselves determined by decisions taken in and between the richer industrialised countries. Not surprisingly, this distinction between internal and external factors has influenced the maimer in which NGOs approach long term development and poverty alleviation ' It began its work in Greece, and only later shifted its emphasis principally to developing countries. ^ Some, however, like the British Red Cross Society or the International Committee of the Red Cross (ICRC), are still predominantly oriented to responding to emergencies. in developing countries. Part of the NGO effort, in Britain as in other industrial countries, is devoted to addressing directly the problems of poverty within developing countries, part is channelled into attempts to influence Britain's (and now the European Community's (EC's)) relevant trade and aid policies. This is done by campaigning, lobbying and increasing public awareness of these external factors through what is termed 'development education'. Since the mid-1960s, this particular aspect of NGO work has become more prominent; indeed some NGOs, such as the World Development Movement and the Centre for World Development Education, were set up to devote themselves exclusively to this aspect of the overall development effort. On the other hand, many NGOs, particularly the smaller ones, devote none or veiy few of their resources to addressing this aspect of the development problem, sometimes because they consider it to be of less importance than helping to promote development within developing countries, in part as a result of discussion with their particular funders and donors.^ Work in the spheres of immediate emergency relief and influencing external policy is thus an important element of the overall activities of most, and all the larger, NGOs. Hence assessment of these activities would need to be incorporated into any comprehensive judgement of the performance and success of NGOs. The focus of this Working Paper, however, is narrower. It is framed exclusively within the context of that part of NGO work directed at alleviating poverty and promoting development through intervention within developing countries, and characterised, typically, by financial and technical support for socio-economic projects and programmes. Its purpose is to attempt to judge the extent to which NGO efforts in this sphere of activity are successful in helping to alleviate poverty. In other words, to assess their impact. It does this in two ways: firstly by outlining a framework for evaluating in some detail one particular type of project undertaken by NGOs, income- generating projects, and secondly by outlining the factors which need to be considered to place this micro-analysis into the wider context. The rest of the Paper is broken down as follows. Section II provides a backdrop to the more technical sections which follow by raising some of the issues which help to illustrate why many British NGOs have tended not to introduce a more widespread and institutionalised policy towards evaluation of their projects, pointing out recent pressures for change, and ending by pointing to a way forward, sensitive to the legitimate concerns of NGOs. Section III outlines, briefly, some of the principles and assumptions which are incorporated into the approach to project and programme evaluation considered in this Paper. Section IV provides a detailed approach to the evaluation of discrete NGO income-generating projects, complemented by Section V which provides a check-list of issues, techniques and questions to assist, where appropriate, the gathering of data required in the evaluation of discrete NGO projects. ^ There are also limits to the manner in which NGOs with charitable status (the vast majority) are permitted under the law to campaign and influence political decisions. For further discussion of this point see Robinson [1990]. Section VI, The Country Assessment, contains a check-list of factors and influences which, it is argued, are necessary to judge the success of particular attempts to alleviate poverty in the framework of the wider problems of poverty and other attempts to address them, at the regional and national level. n. DOUBTS, CONCERNS AND FEARS At the best of times, and in the best of circumstances, evaluation is a delicate and difficuh enterprise. This is not simply because evaluations attempt to assess the worth of things done (or not done) and to suggest ways in which they might be done either better or differently, but because (whether directly or indirectly) the conclusions drawn have a bearing on the work done by individuals and/or the organisations for which they work. What is more, the very nature of evaluation can frequently create tension between evaluator and the project itself. Thus, even if constructively and sympathetically executed, evaluations are often of greatest benefit or interest when they indicate how processes, methods or objectives should or might differ from present practice. Yet such conclusions are precisely those which raise questions about the performance of people and/or institutions responsible for what has been or is currently being done. Institutional concern with (or, sometimes, fear of) the evaluation process is significantly reduced when the decision to evaluate is taken internally by the institution whose projects are being evaluated and is supported by staff from top to bottom; when evaluations are undertaken by people either from within the organisation or employed by it; when the results and conclusions are carefully controlled; and when individuals responsible for the projects under evaluation do not feel their jobs, institutions, or public image will be threatened. Besides these more general concerns there are others, more directly related to aid evaluation, which need to be highlighted. The particular problems of evaluating the development efforts of NGOs can be appreciated first by drawing attention to the difficulties encountered in official aid evaluations, where there is now a history of evaluating projects over a period of 10 to 15 years, and where evaluations are now carried out routinely by all bilateral agencies, including the British Overseas Development Administration (ODA). n.l Problems and Gaps in Evaluation of OfBcial Ajd Although advances have certainly been made in evaluation of official aid, there are still large gaps and uncertainties over method, interpretation and timing of evaluations. In particular, attention can be drawn to the following points. • There is no agreed methodology of evaluation either across different sectors or of particular types of aid interventions. What is more, while there is agreement that evaluations should attempt to judge results against objectives, it is not uncommon in aid interventions for subsequent events to suggest that the objectives initially set might need to be altered, creating an even less firm foundation for deriving durable conclusions. Another problem is that while a literature has been building up of