Accelerated Learning Program for Positive Living and United Service Mid Term Evaluation Review November 7, 2008 This publication was produced for review by the United States Agency for International Development. It was prepared by DevTech Systems, Inc. ACCELERATED LEARNING PROGRAM FOR POSITIVE LIVING AND UNITED SERVICE MID TERM EVALUATION REVIEW Geoff Coyne, Ph.D. Bonnie L. Prince, Ph.D. Eldora Burns Nielson, Ed.D. November 7, 2008 (Final Version) DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. TABLE OF CONTENTS Acknowledgements ............................................................................................................... iii  Acronyms ............................................................................................................................... iv  Definitions ............................................................................................................................... v  EXECUTIVE SUMMARY ......................................................................................................... 1  Summary of Findings ........................................................................................................................... 3  Summary of Recommendations .......................................................................................................... 5  FINDINGS AND RECOMMENDATIONS ................................................................................. 1  Background: Purpose, Objectives, and Methodology of the Mid-Term Review .................................. 1  Findings on the Status of Project Implementation and its Impact ....................................................... 2  Objective 1: Assess Learning Outcomes of ALP Students to Determine the Effectiveness of the Program ..................................................................................................................................... 2  Objective 2: Assess the Performance of ALP Teachers to Determine if Training has a Long-Term Impact on their Teaching Skills and Ability to Provide Quality Instruction ................................ 4  Objective 3: Assess The Impact Of ALPP Activities In Target Counties ........................................... 13  Objective 4. Assess ALP Students By Age To Ensure Appropriate Targeting The Age-Factor In The Determination Of Student Learning Achievement Level ......................................................... 14  Objective 5: Provide a Common Monitoring Tool And Reliable Database to Obtain Information about Quality Implementation of The ALP/ALPP .............................................................................. 15  Objective 6: Scale Up ALP/ALPP to Cater to High Enrollments and Youth Preparation .................. 16  Objective 7: Determine the Contributions of the ALPP Sub-Contracts to the Achievement of the Over-All Goals and Objectives of the Program ....................................................................... 19  Specific Issues for consideration by USAID ...................................................................................... 22  Specific Issues for consideration by Ministry of Education ............................................................... 25  Potential Follow-On Activities ............................................................................................................ 26  APPENDICES ........................................................................................................................ 33  Terms of Reference ........................................................................................................................... 33  Constraints on the Survey ................................................................................................................. 35  Cost effectiveness of the ALPP ......................................................................................................... 36  Technical Discussion of Findings ...................................................................................................... 37  Student Testing Program ............................................................................................................ 37  Teacher Performance ................................................................................................................. 43  Survey Instruments ............................................................................................................................ 55  Student Tests .............................................................................................................................. 55  Lesson Observation Form ........................................................................................................... 69  Interview Instrument: Community Leaders, Chiefs, Members of PTAs, Parents, Education Officers, LRC Staff, Youth Leaders ....................................................................................................... 74  Focus Group Questions for Discussion with Community Leaders, Chiefs, Members of PTAs, Parents, Education Officers, Youth Leaders, LRC Staff ......................................................... 76  Interview Schedule for Community Coordinators at Learning Resource Centers (LRCs) ................ 79  List of Principal Informants ................................................................................................................ 83  REFERENCES ....................................................................................................................... 85  Accelerated Learning Program – PLUS Mid Term Review. Final Report DevTech Systems, Inc. i November 2008 Tables Table 1. Number of students tested by subject, level and county ........................................................... 2  Table 2. Lesson Observations Summary ................................................................................................ 5  Table 3. Classes observed by Class size ................................................................................................ 6  Table 4. Gender of students by program ................................................................................................. 8  Table 5. Lesson Observation Variables with high scores ....................................................................... 8  Table 6. Lesson Observation Variables with moderate or acceptable scores ........................................ 9  Table 7. Lesson Observation Variables with weak or very low scores ................................................... 9  Table 8. Total Beneficiaries of ALPP and ALPP Youth Programs ........................................................ 23  Table 9. Indicative Financing for an Examinations Unit Project ............................................................ 29  Table 10. Cost Effectiveness of the ALPP ............................................................................................ 36  Table 11. Number of students tested by programs ............................................................................... 38  Table 12. Student demographics .......................................................................................................... 38  Table 13. Testing results by program .................................................................................................... 39  Table 14. Results of student testing by subject area ............................................................................ 41  Table 15. Summary of Student Testing Program: Schools and Programs. .......................................... 52  Table 16. Summary of Lesson Observations ........................................................................................ 53  Table 17. Summary of Interviews and Focus Group Activities of Community Representatives and Professional Officers ..................................................................................................................... 54  Photos Photo 1 Evaluation Team in the field ....................................................................................................... 1  Photo 2 Student Testing .......................................................................................................................... 3  Photo 3 Community Focus Group ......................................................................................................... 13  Photo 4 ALPP Youth Class in Language Arts ....................................................................................... 19  Photo 5 Children in crèche while mothers attend THINK Inc. classes .................................................. 21  Accelerated Learning Program – PLUS Mid Term Review Final Report DevTech Systems, Inc. ii November 2008 ACKNOWLEDGEMENTS The conduct of this mid term review was possible only through the advice, cooperation and support of many people. Most are listed in the appendix of this report, but special mention must be made of the following. The team reported directly to Mr. Gib Brown, Basic Education Advisor, USAID/Monrovia. He was a constant source of technical advice and shared his local knowledge of Liberia. He guided the team to relevant background reports, often providing electronic copies. Ms. Margaret K. Sancho-Morris, Education Team Leader, gave additional advice and support. At the Ministry of Education (MOE), the team benefited greatly from an initial meeting with Deputy Minister Hester Williams Catakaw; Professor James Roberts, Deputy Minister, Planning, Research and Development; and Mr. Yonton B. Kesselly Sr, Assistant Minister, Vocational & Technical Education. Mr. B. Cherbutue Quayeson, Director of Research and Planning, MOE was the Focal Point Person for MOE for the Evaluation Team. For field visits to six Counties, he arranged the team’s visits to school sites through the County Education Officers, and accompanied the team throughout the field visits. Mr. Alphonso Sheriff, MOE National Focus Person for the Accelerated Learning Program, advised and organized Workshop #1 to discuss the mid term’s Inception Report. At Creative Associates International, Inc. Dr. Peggy Poling, Chief of Party, ensured that the team had full documentation of the project’s activities. She made herself and her staff available as required, and offered useful advice and support in planning for field and school site visits. Dr. Gail vonHahmann, Acting Chief of Party, also provided invaluable advice and assistance. Mr. Trokon Wayne, Assistant Manager, Supervision and Monitoring, and Mr. Moses Kwalula, Assistant Manager, Training and Materials Development, provided technical advice in planning school site visits, and in furthering the team’s knowledge and understanding of school records and MOE data bases. Ms. Catherine Lumeh was largely responsible for informing Community Coordinators of Learning Resource Centers of the team’s planned activities in Counties, arranging the team’s accommodation in Counties, and assisting with transport arrangements in several counties. Mr. Thomas Nimineh, Community Coordinator of the Monterrado Learning Resource Center, kindly allowed the team the use of the center for work space and workshops. Mr. Alvin Dahn provided daily logistical support throughout the entire period of the team’s review and assisted in the conduct of interviews and focus group work. For logistical advice and assistance, the team is grateful to Ms. Masha Mincic, Air Movement Officer, World Food Programme, who organized a flight to Maryland County. Mr. Mike James of UNMIL assisted in applying for permission to fly with UNMIL. The administration of student tests, and related data entry of test results, was carried out capably and under arduous conditions by Mr. Abba Karnga and Mr. Edwinio Bowman. The cooperation and assistance of County and District Education Officers, School Principals, Teachers, Community Coordinators, and members of Community Groups, is gratefully acknowledged. They gave freely of their time, sharing their years of experience and their considered opinions with the team. Some are named in the List of Contacts. Dr. Geoff Coyne Team Leader and Evaluation Specialist. Dr. Bonnie L. Prince Accelerated Learning Specialist. Dr. Dori Nielson Monitoring and Evaluation Specialist. Accelerated Learning Program – PLUS. Mid Term Review Final Report DevTech Systems, Inc. iii November 2008 ACRONYMS AL Accelerated Learning ALP Accelerated Learning Program ALPP Accelerated Learning Program for Positive Living and United Service Also referred to as ALP - PLUS ALS Accelerated Learning Specialist CAII Creative Associates International, Inc. CEO County Education Office/r COP Chief of Party DEO District Education Office/r EMIS Education Management Information System IP Implementing Partner LRC Learning Resource Center LTTP Liberia Teacher Training Program M & ES Monitoring and Evaluation Specialist MOE Ministry of Education NGO Non-Government Organization NIMAC National Informatics and Management Center NPSCE National Primary School Certificate Examination PERP Primary Education Recovery Program PTA Parent Teacher Association SC - UK Save the Children – United Kingdom SWaP Sector Wide Approach THINK Inc Touching Humanity in Need of Kindness TL Team Leader UNHCR United Nations High Commission for Refugees UNICEF United Nations Children’s Fund USAID United States Agency for International Development WAEC West African Examinations Council Accelerated Learning Program – PLUS Mid Term Review Final Report DevTech Systems, Inc. iv November 2008 DEFINITIONS The Accelerated Learning Program (ALP) was developed for and implemented in Liberia starting in 1998. Its goal is to meet the educational needs of primary school drop-outs, and over-aged children and youth, by enabling them to complete their primary education in three years; it is also hoped to prevent further civil wars by engaging young people in schooling. ALP is implemented in ten counties of Liberia by a variety of Implementing Partners (IPs)/Non-governmental organizations (NGOs), including Creative Associates International, Inc. (CAII). The Accelerated Learning Program for Positive Living and United Service (ALPP) was introduced by CAII in October 2006 under contract to the United States Agency for International Development (USAID). It builds on the ALP and emphasizes the contributions of communities and Parent Teacher Associations (PTAs). It enhances the ALP program through the addition of (i) Youth classes whose curriculum includes a Life Skills component, which is intended to help students better understand important matters about living well, and Service Learning in which students perform service to their schools and communities; (ii) training for PTAs which helps them (a) understand their rightful role in school management, (b) enhance their self confidence, (c) identify and build on community resources to strengthen school programs, and (d) apply for grants from CAII to assist school programs; (iii) Learning Resource Centers (LRCs) in each of six counties which (a) provide in-service workshops, (b) produce learning resource materials, (c) provide a reading room with resource materials to enable teachers, students, education officers, and community members to read and conduct research, (d) provide access to computers to enable computer training and permit access to the internet for research purposes as well as communication through e-mails; (e) provide photocopying and printing facilities for schools and education officers; and (iv) Small Grants scheme to allow PTAs, communities and youth to access funding to expand and enhance schools and informal learning sites. Accelerated Learning Program – PLUS Mid Term Review Final Report DevTech Systems, Inc. v November 2008 This page was left blank intentionally Accelerated Learning Program – PLUS Mid Term Review Final Report DevTech Systems, Inc. vi November 2008 EXECUTIVE SUMMARY The Accelerated Learning Program for Positive Living and United Service (ALPP) commenced in October 2006 and is to continue under its present contract with the United States Agency for International Development (USAID) until October 2009. USAID contracted DevTech Systems, Inc. to conduct a mid term review of the program in order to evaluate (i) the extent to which it was meeting its objectives; (ii) the strengths of the program and its impact on communities; (iii) any issues arising which needed resolution; and (iv) possible future directions of the program and/or an alternative program. The evaluation was carried out between July 14 and November 7, based on (i) a critical review of related reports; (ii) a preparatory workshop to outline the proposed methodology and work schedule and to invite expert appraisal of the plan; (iii) extensive interviews with program administrators at central and county levels; (iv) field visits to each of the six counties in which ALPP is operating where (a) interviews and focus group activities were conducted with Education officers, School Principals, Teachers, and Community Representatives; (b) students of ALP and ALPP were tested in Language Arts and Mathematics at Levels II and III and Grades 3 and 5; and (c) classroom observations of teacher performance and inspection of learning resources were carried out; (v) briefings of USAID officers to discuss findings and recommendations; and (vi) workshop discussion and evaluation with officials of the Ministry of Education (MOE), Participating Partners in the ALP program, and related programs. The review team spent three months in Liberia, one month of which was spent visiting the six counties where ALPP is implemented: Lofa, Bong, Monteserrado, Maryland, Grand Gedeh and Nimba. A total of 18 school sites were visited, covering the educational programs of Accelerated Learning Program Plus (ALPP), Accelerated Learning Program (ALP Regular), and Conventional Primary School Grades 1-6. In total (i) 806 students were tested in Mathematics and Language Arts; (ii) 39 classroom observations were carried out; (iii) 18 school principals were interviewed and/or were members of a Community Focus Group; (iv) seven County Education Officers (CEOs) and District Education Officers (DEOs) were interviewed and/or were members of a Community Focus Group; and (v) five community coordinators were interviewed. In Monrovia, officers in relevant departments of MOE were interviewed: Education Management Information Service (EMIS); Department for Parent Teacher Associations; Accelerated Learning Program Coordinating Unit; Department of Vocational and Technical Education. Officers of Talking Drum Studio and THINK Inc., sub-contractors to the ALPP program, were interviewed, as were officials of the West African Examinations Council (WAEC). Advice and assistance were also obtained from discussions with officers of the Liberia Teacher Training Program (LTTP) and UNICEF. Meetings were held with senior officers of CAII who provided the team with all necessary documentation. Throughout the six counties, there is clear evidence of the overwhelming support for the ALPP Regular and ALPP Youth programs, among students, teachers, school principals, education officers, and community representatives (parents and members of PTAs, Youth leaders, and chiefs). They say that the programs are reasonably resourced, they like the student centered style of teaching, the programs provide basic literacy and numeracy, and the Youth classes with their Life Skills and Service Learning components are helping to make their communities more peaceful and better places to live and work. Almost unanimously they ask for the programs to expand their enrollments, and to reach out further to those communities which do not have ALP or ALPP programs. The programs are allowing over- aged students and youth to complete their primary cycle of education and in some cases to continue on to Junior High School. Accelerated Learning Program – PLUS. Mid Term Review Final Report DevTech Systems, Inc. 1 November 2008 Tests of students at Levels II and III of this year’s program (students who have recently completed one and two years of the program) show that students in general are performing at least as well as those younger students who are following a conventional six year primary cycle of education. Total beneficiaries of the ALPP Regular program were 16,288 students enrolled in 2007- 2008, and 2,649 graduating with a Primary School Certificate in that year. Total enrollments in the ALPP Youth Program were 5,519, with 768 graduating with a Primary School Certificate. Per capita costs of the ALPP, in terms of the targeted number of students are $200 per ALPP student; $124 per THINK Inc. student/beneficiary; and $14 per Talking Drum Studio student. The issue of the future of the whole ALP program (currently supported by ten NGOs) and ALPP is uncertain. Within the MOE, some would phase the ALP into an Adult Literacy Program, keeping the focus on basic literacy and numeracy and including Life Skills and Service Learning in the curriculum. An important element of this approach is the consideration that it would free up resources to the conventional primary cycle of education which are currently in the ALP. Against this is the current age range of the ALP students (10 – 18 years); they fall into the broad category of being too old to fit easily in with younger students of the conventional primary cycle, and are too young to fit easily into adult literacy classes which are also held of a night time. A broader policy question on the future of the ALP and ALPP programs relates to MOE policy, plans and commitments, which are ongoing. Work is continuing on the ALP Policy statement; the policy of Decentralization is still in its early stages; a Sector Wide Approach (SWaP) relating to the working relationship between MOE and NGOs is yet to be investigated; the Education Sector Master Plan 2008-2012 is still under development; and details of the Action Plan for Education For All (EFA), setting out how Liberia will meet its target of all children of school age being in primary school by 2015, are still not clear. Until all of these are clarified, the future role of the ALP and ALPP programs in terms of their efficient and effective contribution to basic education will remain uncertain. Accelerated Learning Program – PLUS Mid Term Review Final Report DevTech Systems, Inc. 2 November 2008