NCEE 2010–4014 U.S. DEpartmENt of EDUCatioN The Effectiveness of a Program to Accelerate Vocabulary Development in Kindergarten (VOCAB) Kindergarten Final Evaluation Report The Effectiveness of a Program to Accelerate Vocabulary Development in Kindergarten (VOCAB) Kindergarten Final Evaluation Report November 2010 Authors: Barbara Goodson, Principal Investigator Abt Associates Inc.* Anne Wolf, Director of Evaluation Abt Associates Inc. Steve Bell, Task 2 Methodological Leader Abt Associates Inc. Herb Turner, Technical consultant ANALYTICA Pamela B. Finney, Regional Educational Laboratory Southeast Research Management Leader SERVE Center at the University of North Carolina at Greensboro *Joan McLaughlin, Original Principal Investigator, Abt Associates Project Officer: Sandra Garcia Institute of Education Sciences NCEE 2010–4014 U.S. Department of Education U.S. Department of Education Arne Duncan Secretary Institute of Education Sciences John Q. Easton Director National Center for Education Evaluation and Regional Assistance Rebecca A. Maynard Commissioner November 2010 This report was prepared for the National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, under contract ED-06C0-0028 with Regional Educational Laboratory Southeast administered by the SERVE Center at the University of North Carolina at Greensboro. IES evaluation reports present objective information on the conditions of implementation and impacts of the programs being evaluated. IES evaluation reports do not include conclusions or recommendations or views with regard to actions policymakers or practitioners should take in light of the findings in the report. This report is in the public domain. Authorization to reproduce it in whole or in part is granted. While permission to reprint this publication is not necessary, the citation should read: Goodson, B., Wolf, A., Bell, S., Turner, H., and Finney, P.B. (2010).The Effectiveness of a Program to Accelerate Vocabulary Development in Kindergarten (VOCAB). (NCEE 2010- 4014). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. This report is available on the Institute of Education Sciences website at http://ncee.ed.gov and the Regional Educational Laboratory Program website at http://edlabs.ed.gov. Alternate Formats Upon request, this report is available in alternate formats, such as Braille, large print, audiotape, or computer diskette. For more information, please contact the Department’s Alternate Format Center at 202-260-9895 or 202-205-8113. ii Disclosure of potential conflict of interest1 None of the authors or other staff involved in the study from Abt, ANALYTICA, Empirical Education, the Regional Educational Laboratory-Southeast, SERVE Center at the University of North Carolina at Greensboro, or the University of Georgia has financial interests that could be affected by the content of this report.1 No one on the nine-member Technical Working Group, convened twice annually by the research team to provide advice and guidance, has financial interests that could be affected by the study findings. 1 Contractors carrying out research and evaluation projects for IES frequently need to obtain expert advice and technical assistance from individuals and entities whose other professional work may not be entirely independent of or separable from the tasks they are carrying out for the IES contractor. Contractors endeavor not to put such individuals or entities in positions in which they could bias the analysis and reporting of results, and their potential conflicts of interest are disclosed. iii CONTENTS SUMMARY...........................................................................................................................................................1   CHAPTER  1:  INTRODUCTION  AND  STUDY  OVERVIEW............................................................................................4   ROLE  OF  VOCABULARY  KNOWLEDGE  IN  READING  COMPREHENSION..........................................................................................4   SELECTING  A  VOCABULARY  INSTRUCTION  PROGRAM..............................................................................................................4   THEORY  OF  CHANGE  FOR  K-­‐PAVE....................................................................................................................................7   EXPERIMENTAL  AND  QUASI-­‐EXPERIMENTAL  EVIDENCE  OF  IMPACTS  OF  VOCABULARY  INSTRUCTION..................................................8   STUDY  TARGETS  THE  MISSISSIPPI  DELTA  REGION  TO  ADDRESS  LOW  STUDENT  READING  ACHIEVEMENT...........................................12   STUDY  DESIGN............................................................................................................................................................13   REPORT  OVERVIEW......................................................................................................................................................15   CHAPTER  2:  STUDY  DESIGN  AND  METHODOLOGY..............................................................................................16   OVERVIEW.................................................................................................................................................................16   SAMPLE  RECRUITMENT  AND  RANDOM  ASSIGNMENT............................................................................................................17   DATA  COLLECTION.......................................................................................................................................................28   ANALYTIC  SAMPLE........................................................................................................................................................40   DATA  ANALYSIS  METHODS.............................................................................................................................................45   CHAPTER  3:  IMPLEMENTATION  OF  THE  INTERVENTION......................................................................................52   OVERVIEW.................................................................................................................................................................52   DESIGN  OF  THE  K-­‐PAVE  PROGRAM.................................................................................................................................52   K-­‐PAVE  PROGRAM  COMPONENTS  AND  TEACHING  STRATEGIES.............................................................................................54   K-­‐PAVE  TEACHER  TRAINING  AND  SUPPORT.......................................................................................................................57   PROCEDURE  FOR  ASSESSING  FIDELITY  OF  IMPLEMENTATION  OF  K-­‐PAVE.................................................................................60   FINDINGS  ON  FIDELITY  OF  K-­‐PAVE  IMPLEMENTATION  IN  THE  INTERVENTION  CLASSROOMS.........................................................65   SUMMARY  OF  FINDINGS  AND  CONCLUDING  OBSERVATIONS..................................................................................................68   CHAPTER  4:  IMPACT  RESULTS.............................................................................................................................69   SUMMARY  OF  K-­‐PAVE  IMPACTS....................................................................................................................................69   RESEARCH  QUESTIONS..................................................................................................................................................70   IMPACTS  ON  KINDERGARTEN  STUDENTS’  EXPRESSIVE  VOCABULARY.........................................................................................70   IMPACTS  ON  KINDERGARTEN  STUDENTS’  ACADEMIC  KNOWLEDGE  AND  LISTENING  COMPREHENSION.............................................71   IMPACTS  ON  CLASSROOM  INSTRUCTION...........................................................................................................................73   CHAPTER  5:  SUMMARY  OF  FINDINGS  AND  STUDY  LIMITATIONS........................................................................78   EFFECT  OF  K-­‐PAVE  ON  EXPRESSIVE  VOCABULARY..............................................................................................................78   EFFECT  OF  K-­‐PAVE  ON  OTHER  VOCABULARY-­‐RELATED  LITERACY  SKILLS..................................................................................78   EFFECT  OF  K-­‐PAVE  ON  CLASSROOM  INSTRUCTION.............................................................................................................78   STUDY  PARAMETERS.....................................................................................................................................................79   STUDY  LIMITATIONS.....................................................................................................................................................80   APPENDIX  A.  MISSISSIPPI  COUNTIES  WITH  STUDY  SCHOOLS,  BY  COUNTY..........................................................81   APPENDIX  B.  STATISTICAL  POWER  ANALYSIS......................................................................................................82   STATISTICAL  POWER  FOR  DETECTING  IMPACTS  ON  STUDENTS................................................................................................82   ACTUAL  MINIMUM  DETECTABLE  EFFECT  SIZES  FOR  STUDENTS’  EXPRESSIVE  VOCABULARY..........................................................84   STATISTICAL  POWER  FOR  DETECTING  IMPACTS  ON  CLASSROOM  INSTRUCTION...........................................................................85   iv ACTUAL  MINIMUM  DETECTABLE  EFFECT  SIZES  FOR  CLASSROOM  INSTRUCTION  OUTCOMES...........................................................86   APPENDIX  C.  RANDOM  ASSIGNMENT.................................................................................................................88   MATCHING  OF  SCHOOLS  WITHIN  BLOCKS  FOR  RANDOM  ASSIGNMENT.....................................................................................88   PROCESS  OF  RANDOM  ASSIGNMENT:  SEQUENCE  GENERATION...............................................................................................89   APPENDIX  D.  RECRUITMENT  AND  RANDOM  SELECTION  OF  THE  STUDENT  SAMPLE............................................93   APPENDIX  E.  COMPARISON  OF  STUDENTS  MISSING  AND  NOT  MISSING  BASELINE  ASSESSMENT........................94   APPENDIX  F.  CLASSROOM  OBSERVATION  MEASURES  FOR  IMPACT  EVALUATION...............................................95   CLASSROOM  ASSESSMENT  SCORING  SYSTEM......................................................................................................................95   READ  ALOUD  PROFILE–KINDERGARTEN...........................................................................................................................98   VOCABULARY  RECORD..................................................................................................................................................99   CREATION  OF  VOCABULARY  AND  COMPREHENSION  SUPPORT  COMPOSITE.............................................................................100   APPENDIX  G.  TEACHER  SURVEY........................................................................................................................105   APPENDIX  H.  K-­‐PAVE  FIDELITY  OBSERVER  HANDBOOK  AND  TRAINING  FIDELITY  CHECKLIST.............................109   KINDERGARTEN  PAVED  FOR  SUCCESS  (K-­‐PAVE)  FIDELITY  OBSERVATIONS............................................................................109   TRAINING  FIDELITY  CHECKLIST  DETAILED  DIRECTIONS  AND  IMPLICATIONS  FOR  NOTE  TAKING......................................................110   FORM  COMPLETION−TRAINING  FIDELITY  CHECKLIST.........................................................................................................115   TEACHER  OBSERVATION  FOLLOW-­‐UP  MEETING  PROTOCOL—DETAILED  DESCRIPTION..............................................................119   TEACHER  OBSERVATION..............................................................................................................................................122   FOLLOW-­‐UP  MEETING  PROTOCOL.................................................................................................................................122   FREQUENTLY  ASKED  QUESTIONS...................................................................................................................................124   APPENDIX  I.  DATA  COLLECTION  PROCEDURES..................................................................................................125   PROTOCOL  FOR  CHILD  ASSESSMENTS:  QUICK  REFERENCE...................................................................................................125   PROTOCOL  FOR  CLASSROOM  OBSERVATIONS..................................................................................................................126   APPENDIX  J.  DATA  QUALITY  ASSURANCE  PROCEDURES....................................................................................128   APPENDIX  K.  MODEL  SPECIFICATIONS..............................................................................................................130   THREE-­‐LEVEL  MODEL  USED  TO  ESTIMATE  IMPACTS  ON  KINDERGARTEN  STUDENTS  (FOR  SINGLE  OUTCOME  MEASURES)....................130   MODELS  SPECIFICATION  FOR  GLOBAL  F-­‐TEST  OF  JOINT  IMPACT  ON  MULTIPLE  STUDENT  OUTCOMES  WITHIN  A  DOMAIN...................132   TWO-­‐LEVEL  MODEL  USED  TO  ESTIMATE  IMPACTS  ON  CLASSROOM  INSTRUCTION  (FOR  SINGLE  OUTCOME  MEASURES)......................134   MODELS  SPECIFICATION  FOR  GLOBAL  F-­‐TEST  OF  JOINT  IMPACT  ON  MULTIPLE  CLASSROOM  INSTRUCTION     OUTCOMES  WITHIN  A  DOMAIN..................................................................................................................................136   APPENDIX  L.  FLOWCHART  ILLUSTRATING  SAMPLE  ATTRITION  FROM  DATA  COLLECTION.................................139   APPENDIX  M.  MISSING  DATA  IMPUTATION......................................................................................................140   DUMMY  VARIABLE  ADJUSTMENT  FOR  MISSING  COVARIATES................................................................................................140   SINGLE  STOCHASTIC  REGRESSION  IMPUTATION  FOR  MISSING  PRETEST  AND  POSTTEST  DATA.......................................................142   APPENDIX  N.  SENSITIVITY  ANALYSES................................................................................................................146   IMPACTS  ON  STUDENTS..............................................................................................................................................146   IMPACTS  ON  CLASSROOM  INSTRUCTION.........................................................................................................................153   APPENDIX  O.  SCHOOL,  TEACHER,  AND  STUDENT  COVARIATES.........................................................................158   v APPENDIX  P.  LIST  OF  K-­‐PAVE  MATERIALS  PROVIDED  TO  TEACHERS..................................................................161   APPENDIX  Q.  SAMPLE  WEEKLY  UNIT  FROM  K-­‐PAVE  PROGRAM........................................................................162   TRANSPORTATION......................................................................................................................................................162   CAR  TALK................................................................................................................................................................163   PAIRED  WORDS  FOR  TRANSPORTATION.........................................................................................................................163   EXTENSION  ACTIVITIES................................................................................................................................................164   APPENDIX  R.  LIST  OF  THE  240  K-­‐PAVE  TARGET  WORDS....................................................................................166   APPENDIX  S.  K-­‐PAVE  TEACHER  TRAINING  AGENDA..........................................................................................168   DAY  1.....................................................................................................................................................................168   DAY  2.....................................................................................................................................................................168   K-­‐PAVE  ASSISTANT  TEACHER  SCHEDULE  (DAY  2  ONLY)....................................................................................................168   APPENDIX  T.  K-­‐PAVE  TEACHER  PHONE  FOLLOW-­‐UP  AGENDA...........................................................................170   I.  GENERAL  PROBLEMS  (ASK  EACH  FOLLOW-­‐UP)...............................................................................................................170   II.  PROGRAM  AREAS...................................................................................................................................................170   III.  WHAT  CAN  WE  DO  TO  HELP  YOU  WITH  THE  PROGRAM?................................................................................................173   IV.  REMINDER  OF  WHEN  NEXT  PHONE  CALL  WILL  BE..........................................................................................................173   APPENDIX  U.  SAMPLE  MEANS  AND  STANDARD  DEVIATIONS  FOR  STUDENT  AND  CLASSROOM  OUTCOME   MEASURES,  BY  INTERVENTION  STATUS......................................................................................................174   APPENDIX  V.  CHECKING  MODEL  ASSUMPTIONS...............................................................................................175   MODEL  EXAMINING  IMPACTS  ON  STUDENTS’  EXPRESSIVE  VOCABULARY  (EVT–2)...................................................................175   MODELS  EXAMINING  IMPACTS  ON  CLASSROOM  INSTRUCTION.............................................................................................178   PROPORTION  OF  CLASSROOM  OBSERVATION  CYCLES  SPENT  ON  NONVOCABULARY  LITERACY  INSTRUCTION....................................183   APPENDIX  W.  TRANSLATING  IMPACTS  ON  STUDENTS  INTO  AGE-­‐EQUIVALENT  DIFFERENCES  IN  POSTTEST   OUTCOMES...................................................................................................................................................186   REFERENCES.....................................................................................................................................................188   BOXES Box 2.1 Summary of kindergarten evaluation design.................................................................................16   Box F1. Read Aloud Profile–Kindergarten coding form............................................................................99   Box F2. Vocabulary Record coding form.................................................................................................102   FIGURES Figure 2.1. District and school recruitment process and timeline...............................................................20   Figure 2.2. Outcome of random assignment of schools and random selection of classrooms in the study sample..............................................................................................................................23   Figure 2.3 Flow of students through the study............................................................................................39   Figure D1. Recruitment and random selection of the student sample........................................................93   Figure V1. Studentized residuals at the student level plotted against a normal distribution....................176   Figure V2. Raw residuals versus normal distribution at classroom and school levels.............................176   vi Figure V3. Studentized residuals at the student level plotted against EVT posttest score for students in treatment (red) and control group (blue) schools.................................................177   Figure V4. Studentized residuals at the classroom level (left panel) and school level (right panel) plotted versus EVT posttest score for students in treatment (red) and control group (blue) schools.....................................................................................................................................177   Figure V5. Plot of residuals versus a normal distribution at the classroom and school levels, from the model testing the impact on vocabulary and comprehension support.....................178   Figure V6. Residuals at the classroom level and school level versus vocabulary and comprehension support posttest score for students in treatment (red) and control group (blue) schools........179   Figure V7. Plot of residuals versus a normal distribution at the classroom and school levels, from the model testing the impact on instructional support...................................................180   Figure V8. Residuals at the classroom level and school level versus instructional support posttest score for students in treatment (red) and control group (blue) schools.....................181   Figure V9. Plot of residuals versus a normal distribution at the classroom and school levels, from the model testing the impact on emotional support.......................................................182   Figure V10. Residuals at the classroom level and school level versus emotional support posttest score for students in treatment (red) and control group (blue) schools.....................183   Figure V11. Plot of residuals versus a normal distribution at the classroom and school levels, from the model testing the impact on nonvocabulary literacy instruction..............................184   Figure V12. Residuals at the classroom level and school level versus nonvocabulary literacy instruction for students in treatment (red) and control group (blue) schools..........................185   MAP Map A1. Mississippi counties with study schools, by county....................................................................81   TABLES Table 2.1 Comparison of schools that agreed to participate and all eligible schools.................................25   Table 2.2 Description of school sample......................................................................................................26   Table 2.3 Reading programs in place at baseline in intervention and control schools...............................28   Table 2.4 Timeline of study activities: data collection and intervention implementation..........................29   Table 2.5 Child measures, outcome variables, and designations................................................................30   Table 2.7 Response rates for data collected from schools, classrooms, teachers, and students for intervention and control groups (percent).................................................................................38   Table 2.8 Nonparticipants in intervention and control groups....................................................................40   Table 2.9 Comparison of initial sample at randomization and final analytic sample.................................41   Table 2.10 Characteristics of teachers in the analytic sample, by intervention condition..........................42   Table 2.11 Characteristics of students in the analytic sample, by intervention condition..........................43   Table 2.12 Baseline pretest scores on student outcomes for intervention and control groups...................44   Table 2.13 Baseline measures of classroom instruction for intervention and control groups (n = 64 schools, 128 classrooms)..........................................................................................................45   vii Table 2.14 Primary research question, outcome variable, and need for adjustment for multiple tests.......48   Table 2.16 Number and percentage of students, teachers, and schools missing covariate data.................51   Table 2.17 Number and percentage of students missing either pretest or posttest assessment, by intervention and control group..................................................................................................51   Table 3.1 Methodologies for addressing questions on implementation of K-PAVE..................................53   Table 3.2 Schedule of K-PAVE teacher training and support activities.....................................................58   Table 3.3 Methodology for assessing fidelity of K-PAVE training and support for teachers in intervention schools..............................................................................................................61   Table 3.4 Coding K-PAVE classroom fidelity: training fidelity checklist items and coding protocol.......63   Table 3.5 Proportion of intervention teachers participating in K-PAVE training and support...................65   Table 3.6 Participation of intervention teachers in K-PAVE teacher training and support activities.........66   Table 3.7 Number of teaching strategies implemented...............................................................................66   Table 3.8 Presence of instructional strategies by K-PAVE program components......................................67   Table 4.1 Estimated regression-adjusted impact of K-PAVE on kindergarten students’ expressive vocabulary.................................................................................................................................71   Table 4.2 Estimated regression-adjusted impact of K-PAVE on kindergarten students’ listening comprehension and academic knowledge.................................................................................73   Table 4.3 Estimated regression-adjusted impacts of K-PAVE on classroom instruction in kindergarten..........................................................................................................................74   Table 4.4 Sample control group means and standard deviations for four components of vocabulary and comprehension support composite measure (n = 128 classrooms)....................................75   Table 4.5 Estimated regression-adjusted impact of K-PAVE on amount of literacy instruction in areas other than vocabulary and comprehension in kindergarten.............................................77   Table B1. Power analysis summary: minimum detectable effect sizes for student outcomes, by number of schools................................................................................................................83   Table B2. Comparison of assumed and observed factors related to minimum detectable effect size for the Expressive Vocabulary Test–2 posttest.........................................................................84   Table B3. Power analysis summary: minimum detectable effect sizes for classroom instruction outcomes, by number of schools.............................................................................86   Table B4. Minimum detectable effect sizes and estimated impacts in the observed sample for classroom instruction outcomes..........................................................................................86   Table B5. Comparison of assumed and observed factors related to minimum detectable effect sizes for classroom instruction outcomes.......................................................................86   Table C1 Random assignment for a hypothetical list of Reading First Schools, ordered based on school characteristics...........................................................................................................91   Table E1. Characteristics of students missing and not missing baseline assessment.................................94   Table M1. Missing data on school covariates in treatment and control samples......................................141   Table M2. Missing data on student pretest and posttest assessments, for treatment and control groups (596 treatment students, 700 control students)...............................................142   Table N1. Estimated impact on kindergarten students’ expressive vocabulary (EVT–2) in models fit for sensitivity analysis compared with final impact model in Chapter 4...............148   viii Table N2. Results of sensitivity analysis conducted on joint test of K-PAVE impact on academic knowledge and listening comprehension................................................................................149   Table N3. Estimated impact on kindergarten students’ listening comprehension in models fit for sensitivity analysis compared with final impact model presented in Chapter 4.....................151   Table N4. Estimated impact on kindergarten students’ academic knowledge in models fit for sensitivity analysis compared with final impact model presented in Chapter 4.....................152   Table N5. Results of sensitivity analysis conducted on joint test of K-PAVE impact on vocabulary and comprehension support, instructional support, and emotional support.........154   Table N6. Results of sensitivity analyses of estimated K-PAVE impact on vocabulary and comprehension support...........................................................................................................156   Table N7. Results of sensitivity analyses of estimated K-PAVE impact on instructional support...........156   Table N8. Results of sensitivity analyses of estimated K-PAVE impact on emotional support...............157   Table N9. Results of sensitivity analyses of estimated K-PAVE impact on proportion of cycles spent on nonvocabulary literacy instruction...............................................................157   Table O1. Student-level covariates...........................................................................................................158   Table O2. Teacher-level covariates...........................................................................................................159   Table O3. School covariates......................................................................................................................160   Table U1. Sample intervention and control group means for student outcome measures........................174   Table U2. Sample intervention and control group means for classroom instruction outcome measures...................................................................................................................174   ix