EdData II DEP/AME: Philippines Analytic Support Services for Early Grade Reading (PhilEd Data II) Component 2: Early Grade Reading Assessment Results: A cross-language look at MTB-MLE implementation in the Philippines EdData II Technical and Managerial Assistance, Task Number 15 Contract Number: AID-492-M-12-00001 July 30, 2014 This publication was produced for review by the United States Agency for International Development. It was prepared by RTI International. Philippines EGRA 2014 – Final Report (4 languages) i DEP/AME: Philippines Analytic Support Services for Early Grade Reading (PhilEd Data II) Component 2: Early Grade Reading Assessment Results: A cross-language look at MTB-MLE implementation in the Philippines Education Data for Decision Making (EdData II) Task Order No. 15 Prepared for USAID/Washington Mitch Kirby Senior Education Advisor, Asia Bureau Contracting Officer’s Technical Representative Data for Education Programming/Asia and Middle East 1300 Pennsylvania Avenue NW Washington, DC 20523 Prepared by Sarah Pouezevara, Joseph DeStefano, Chris Cummiskey and Jennifer Pressley 3040 Cornwallis Road Post Office Box 12194 Research Triangle Park, NC 27709-2194 RTI International is a trade name of Research Triangle Institute. The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Philippines EGRA 2014 – Final Report (4 languages) ii Table of Contents List of Tables ........................................................................................................... iv List of Figures ......................................................................................................... iv Abbreviations .......................................................................................................... vi Acknowledgments.................................................................................................. vii Map of the Philippines and regions of data collection ....................................... viii I. Introduction ........................................................................................................... 1 A. Background ........................................................................................................ 1 B. DepED’s MTB-MLE ......................................................................................... 2 C. Study design and methodology .......................................................................... 7 II. Findings ....................................................................................................... 11 A. Is MTB-MLE being implemented? ................................................................. 11 B. What are the predominant teaching methods being used? ............................... 16 C. How well are children learning how to read in the MT? ................................. 20 D. How well are children acquiring L2 and L3? .................................................. 39 E. What factors are associated with reading skills? ............................................. 46 III. Summary and Conclusions .............................................................................. 51 A. Summary of EGRA results by language .......................................................... 51 B. Limitations ....................................................................................................... 58 C. Conclusions ...................................................................................................... 61 D. Recommendations ........................................................................................... 62 Annex A: Summary of results from 2013 EGRA study ....................................... 66 Annex B: MTB-MLE Index ...................................................................................... 71 Annex C: Sampling methodology ......................................................................... 74 Philippines EGRA 2014 – Final Report (4 languages) iii List of Tables Table 1: Information on languages of 2014 EGRAs ........................................................ 2 Table 2: EGRA instrument subtests in the Philippines .................................................... 8 Table 3: Study sample .................................................................................................... 10 Table 4: National curriculum standards related to reading ............................................ 11 Table 6: Summary of select indicators related to MTB-MLE implementation, by language and grade .......................................................................................... 19 Table 7: Correlations between subtests .......................................................................... 32 Table 8: Reading accuracy: Percent correct out of total attempted ............................... 35 Table 9: Minimum number of words attempted for each comprehension question, by language....................................................................................... 35 Table 10 ORF and comprehension.................................................................................. 38 Table 11: G2 teacher expectations for Filipino reading ................................................... 39 Table 12: Correlations between subtests .......................................................................... 41 Table 13: Summary of EGRA results: Average by subtest and grade ............................. 44 Table 14: Summary of EGRA results: Proportion of zero scores by subtest and grade ................................................................................................................. 45 Table 15: Select indicators for Cebuano .......................................................................... 52 Table 16: Select indicators for Hiligaynon ...................................................................... 53 Table 17: Select indicators for Ilokano ............................................................................ 55 Table 18: Summary of key indicators for Maguindanaoan .............................................. 57 Table 19: Breakdown of classroom sample by type of classroom ................................... 61 Table 20: Reading fluency and comprehension as benchmarks ...................................... 65 Table C-1. Number of Excluded schools from the 2013 BIES school list and reasons why schools were excluded. ............................................................................ 74 Table C-2: Sample methodology for the 2014-Four Region EGRA-SSME............................ 75 Table C-5: Counts by Region-Grade of the completed assessments/questionnaires at the student level. .............................................................................................. 76 List of Figures Figure 1: Scores on the MTB-MLE Implementation Index—Teachers .......................... 13 Figure 2: Scores on the MTB-MLE Implementation Index—Materials ......................... 14 Figure 3: Scores on the MTB-MLE Implementation Index—Instruction ....................... 15 Figure 4: Pupil language activity during MT lesson ....................................................... 17 Figure 5: Classroom configuration during MT lesson .................................................... 18 Table 5: Percent of classroom observation where no instruction was taking place ................................................................................................................. 18 Figure 6: Average percent correct responses on PA and LC tasks .......................................... 21 Figure 7: Scores on listening comprehension and phonemic awareness exercises, by grade and language ..................................................................................... 21 Figure 8: Phonemic awareness: Distribution of scores by grade and language .............. 22 Figure 9: Listening comprehension: Distribution by grade and language ...................... 23 Figure 10: Letter-sound identification scores, by region and grade .................................. 26 Figure 11: Letter-sound identification: Distribution of scores, by grade and language ........................................................................................................... 26 Figure 12: Correct non-words per minute: Average scores by region and grade, with and without zero scores............................................................................ 28 Figure 13: Non-word reading: Distribution of scores, by grade and language ................. 29 Philippines EGRA 2014 – Final Report (4 languages) iv Figure 14: Correct familiar words per minute: Average scores by region and grade, with and without zero scores ................................................................. 30 Figure 15: Familiar word reading: Distribution of scores, by grade and language ........... 31 Figure 16: Oral reading fluency (with and without zero scores), by language and grade ................................................................................................................. 33 Figure 17: Oral reading fluency: Distribution of scores, by grade and language .............. 34 Figure 18: Reading comprehension: Average percent correct by language and grade ................................................................................................................. 36 Figure 19: Distribution of comprehension scores (percent correct out of comprehension questions attempted) by language and grade .......................... 37 Figure 20: Comprehension and oral reading fluency ........................................................ 38 Figure 21: Filipino—Proportion of zero scores................................................................. 40 Figure 22: Filipino familiar word reading: Correct words per minute by language ......... 40 Figure 23: Teacher expectations for English reading acquisition ..................................... 42 Figure 24: English vocabulary: Percent zero scores by language ..................................... 43 Figure 25: English vocabulary: Percent of correct responses by language ....................... 43 Figure 26: Summary of differences in performance by gender, across languages and subtests (percentage point difference, girls-boys) ..................................... 47 Figure 27: Differences in oral reading associated with home factors ............................... 48 Figure 28: Differences in average oral reading fluency associated with school- related factors ................................................................................................... 49 Figure 29: Odds ratios for variables associated with being in the top 25% of readers .............................................................................................................. 50 Figure A-1: Comparison of Filipino scores across regions ................................................. 66 Figure A-2: Comparison of English scores across regions .................................................. 66 Philippines EGRA 2014 – Final Report (4 languages) v Abbreviations ARMM Autonomous Region of Muslim Mindanao cfwpm correct familiar words per minute clspm correct letter sounds per minute cnwpm correct non-words per minute cwpm correct words per minute DepED Department of Education EdData II Education Data for Decision Making EGRA early grade reading assessment G grade IRR inter-rater reliability KFW Filipino language commission LC learning comprehension MT mother tongue MTB-MLE mother-tongue based, multilingual education OFW overseas foreign worker PA phonemic awareness PhilEd Data Philippines Analytic Support Services for Early Grade Reading Phil-IRI Philippines Informal Reading Inventory Q quarter SES socio-economic status USAID United States Agency for International Development Philippines EGRA 2014 – Final Report (4 languages) vi Acknowledgments This study was carried out under the auspices of USAID’s Data for Education Planning in Asia and the Middle East (DEP/AME) task order to the Education Data for Decision Making (EdData II) Blanket Purchase Agreement for which RTI International is the contractor. The work in the Philippines however could not have been completed without the interest, support and commitment of the Education Office of USAID/Philippines, in particular Robert Burch, Mirshariff Tillah and Marie Antoinette Reyes-Hayles. Likewise, the Philippines Department of Education provided the leadership and committed the time and effort necessary to develop the instruments, collect the data and analyze and interpret the results of this study. In particular we wish to acknowledge Undersecretary for Programs and Projects Dr. Dina S. Ocampo and the staff within her office who are committed to improving reading outcomes in the Philippines. This especially includes Dr. Marilyn D. Dimaano, Director of the Bureau of Elementary Education (BEE), Dr. Rosalina J. Villaneza, Focal Person for MTB-MLE in BEE, and Jocelyn Tuguinayo, Senior Education Program Specialist in BEE. The Regional Directors and other Department of Education staff at the regional level in Regions I, VI, VII and ARMM made invaluable contributions to the development of assessment and data collection instruments in each regional language and provided the person-power and logistical support necessary to organize all the field data collection in their respective regions, in particular Dina Bonao, Nerio Benito Eseo, Nena Meñoza and Saada Tubing. We also thank the Regional Directors Tersita M. Velasco, John Arnold S. Siena, Carmelita T. Dulangon, and Jamar M. Kulayan. In addition to the contributions of DepED, the field work would not have been possible without the collaboration of our partner Taylor Nelson Sofres (TNS), in particular Yazmin Tolentino, Joel Flores and Mao Ocampo provided leadership on the part of TNS in the development of the instruments, training of assessors and management of field operations. TNS field staff gamely traveled far and wide to supervise and support data collection in the four regions. Our other partner on all the EdData II work in the Philippines has been SEAMEO INNOTECH. For this study SEAMEO helped organize the final policy dialogue workshop. Lastly, and perhaps most importantly, we acknowledge the tireless efforts of Rufino Jamilano, RTI International’s in-country task coordinator for all EdData II activities. Rufino’s constant diligence and commitment made it possible to collaborate productively with all the players involved in this and previous activities. Philippines EGRA 2014 – Final Report (4 languages) vii Map of the Philippines and regions of data collection Shaded areas designate the areas where data collection for this report took place. The region boundaries and areas where the languages are spoken may be different than the boundaries represented on this map. Philippines EGRA 2014 – Final Report (4 languages) viii I. Introduction A. Background To read and understand a simple text is one of the most fundamental skills a child can learn; in 2014 it should be the responsibility of a functioning primary school system to provide at least sufficient reading skills to the children it serves. Evidence indicates that learning to read both early and at a sufficient rate are essential for learning to read well, and consequently understanding what is read. Acquiring literacy becomes more difficult as learners grow older; those who do not learn to read in the first few grades are more likely to repeat and eventually drop out (Early Grade Reading Assessment [EGRA] Toolkit, RTI International, 2009). The Government of the Philippines has placed high priority on improving literacy and is currently undertaking comprehensive reform initiatives including curricular reforms and implementation of a mother-tongue based multilingual education (MTB-MLE) program in 19 national languages. In support of these reforms, the United States Agency for International Development (USAID)/Philippines and the Philippines Department of Education (DepED) initiated the USAID-funded Strengthening Information for Education Policy, Planning, and Management in the Philippines project (PhilEd Data I) in April 2012. During 2012 and 2013, PhilEd Data provided technical assistance and support to several activities, one of which was the implementation of a national, sample-based evaluation of reading in the early grades of elementary education. The principal outcomes of this component were:  EGRA instruments were adapted to English (Grade 3), Filipino (Grade 3), and Ilokano (Grade 1).  The instruments were used to collect data from a nationally representative sample in Grade 3 (6 regions, 2,400 students) and a regionally-representative sample in Grade 1 (1 region, 500 students).  Results were presented to and discussed with a range of stakeholders in July 2013. The results provided baseline data on Grade 3 students’ reading achievement and language readiness under the previous (prior to adopting mother tongue-based instruction) national curriculum. PhilEd Data I also explored the relationship of these skills to selected factors in students’ school and home environment. This evidence can be used to inform policy and the implementation of appropriate interventions in schools and classrooms across the country, as well as to monitor the effect of MTB-MLE in years to come. The regional language EGRA for Ilokano was the first rigorous and large-scale assessment of student learning in one of the mother-tongue languages. The data on Grade 1 progress towards learning to read in the mother tongue is a starting point from which DepED may develop benchmarks and standards for reading progress in the mother tongue and improve teaching methods and materials based on evidence of the effectiveness of existing inputs. It is in this context that the 2014 Philippines EGRA study (PhilEd Data II) is expanding the knowledge base for the MTB-MLE program by conducting a comparable exercise in three additional mother-tongue languages—Cebuano1, Hiligaynon, and Maguindanaoan (Table 1). Another representative set of data was collected for Ilokano, which will provide additional 1 The local name is Sinugbuanong Binisaya. For convenience, we will refer to the language as Cebuano. Philippines EGRA 2014 – Final Report (4 languages) 1 information on whether quality of MT instruction is improving with implementation experience, and how much children improve their reading skills from Grade 1 to Grade 2. The 2014 study used the same EGRA instrument as 2013—with appropriate adaptions—to observe and measure reading skills in five key areas:  Alphabetic principle  Phonemic awareness  Vocabulary  Fluency  Comprehension The 2014 fieldwork is sponsored by USAID/Washington under the Education Data for Decision-Making (EdData II) program, and technical assistance is provided by RTI International and a local survey research partner, TNS Global2. Table 1: Information on languages of 2014 EGRAs Region # Region Name Language Region population** Est. speakers nationwide*** ARMM Autonomous Maguindanaoan 3,256,140 1,100,000 Region in Muslim Mindanao I Ilocos (Northern Ilokano 4,748,372 6,920,000 Luzon) VI Western Visayas Hiligaynon 7,102,438 5,770,000 VII Central Visayas Sinugbuanong 6,800,180 15,800,000 Binisaya or Cebuano ** Census data from 2010, National Statistical Coordination Board. http://www.nscb.gov.ph/secstat/d_popn.asp *** Data from www.ethnologue.com, based on 2000 census or SIL data B. DepED’s MTB-MLE The current MTB-MLE policy follows decades of experimentation with different language policies and pilot programs each aiming to find the right balance between protecting and valuing indigenous languages (mother tongues), while also allowing Filipinos from across the nation to communicate using a common language (Filipino) and at the same time creating opportunities for international opportunity through development of fluency in an international language (English).3 These experiments revealed varied yet generally positive results supporting the notion that children learn best when instruction is in the language they are most familiar with, while also showing this does not inhibit their ability to progressively learn other languages. In July 2009 DepED issued Order No. 74: “Institutionalizing Mother Tongue-Based Multilingual Education (MTB-MLE).” The Order requested the use of the learner’s mother 2 For more information, contact Sarah Pouezevara, RTI International ([email protected]). 3 For example, First Iloilo Experiment (1948–1954); the Cebu Experiment (pre-1960s); the Antique Experiment (1952); the First Rizal Experiment (1953–1959); another Rizal experiment (1960–1966); the First Language Component Bridging Program (FLC-BP) Pilot Project in Ifugao (1986–1993); the Lubuagan Multilingual Education Program (1998–); the Lingua Franca Project (1999–2001); the Culture-Responsive Curriculum for Indigenous People–Third Elementary Education Project (2003–2007); the Double Exposure in Mathematics Initiative of Region IV-B (2004–2007); and others. Philippines EGRA 2014 – Final Report (4 languages) 2