Abstract  Title  Page         Title:  Examining  Variation  in  Achievement  Impacts  Across  the  KIPP  Network  of  Charter  Schools     Authors  and  Affiliations:     Christina  Clark  Tuttle   Mathematica  Policy  Research     Philip  Gleason   Mathematica  Policy  Research     Joshua  Furgeson   Mathematica  Policy  Research                                               SREE  Spring  2012  Conference  Abstract:  KIPP  i3  Evaluation,  Invited  Symposium Abstract  Body       Background  /  Context:     Description  of  prior  research  and  its  intellectual  context.     Since  its  inception  in  the  mid-­‐1990s,  KIPP  has  grown  from  two  core  middle  schools  to  a  nationwide   network  of  109  charter  schools  in  20  states  and  the  District  of  Columbia  as  of  the  2011–2012  school   year.  KIPP  now  serves  over  32,000  students  at  the  elementary,  middle,  and  high  school  levels  and  plans   to  open  approximately  50  new  schools  over  the  next  three  years.  In  2010,  the  KIPP  Foundation  was   awarded  a  five-­‐year,  $50  million  Investing  in  Innovation  (i3)  scale-­‐up  grant  by  the  U.S.  Department  of   Education’s  Office  of  Innovation  and  Improvement.  The  award  was  one  of  four  i3  scale-­‐up  grants   funding  expansion  of  programs  demonstrating  strong  evidence  of  previous  effectiveness  in  improving   student  achievement  and  educational  attainment.*  The  KIPP  Foundation  plans  to  use  the  i3  grant  to   scale  up  its  program  while  sustaining  KIPP’s  positive  impacts.     Though  the  average  effects  of  charter  schools  overall  are  mixed,  the  consistently  positive  effects  of  KIPP   charter  schools  suggest  that  their  model  may  be  worth  replicating.  Recent  studies  of  KIPP  schools  in  the   San  Francisco  Bay  Area  and  Lynn,  Massachusetts—using  propensity  score  matching  and  a  lottery-­‐based   design,  respectively—found  large  and  statistically  significant  impacts  in  both  math  and  reading,  with   effect  sizes  ranging  from  0.12  to  0.68  standard  deviation  units  in  reading  and  from  0.19  to  0.88  standard   deviation  units  in  math  for  one  year  of  KIPP  instruction  (Woodworth  et  al.  2008;  Angrist  et  al.  2010).  A   2010  Mathematica  study  of  22  KIPP  middle  schools  estimated  impacts  after  three  years  (Tuttle  et  al.   2010).  By  Year  3,  half  of  the  KIPP  schools  in  the  sample  produced  math  impacts  of  0.48  standard   deviations  or  more  (representing  an  estimated  1.2  years  of  additional  instruction),  and  half  of  the  KIPP   schools   in   the   sample   produced   three-­‐year   reading   effects   of   0.28   standard   deviations   or   more   (representing  an  estimated  0.9  year  of  additional  instruction).       However,  to  date,  no  rigorous  research  has  been  published  on  KIPP  schools  at  the  elementary  or  high   school  levels,  which  represent  the  majority  (41  and  27  percent,  respectively)  of  new  school  openings   over  the  past  five  years,  nor  have  any  existing  studies  compared  impacts  over  time  as  the  network  has   grown.       Purpose  /  Objective  /  Research  Question  /  Focus  of  Study:   Description  of  the  focus  of  the  research.     As  a  condition  of  its  i3  grant,  KIPP  contracted  with  an  independent  evaluator  (Mathematica)  to  address  a   key  research  question:  does  KIPP  maintain  its  demonstrated  effectiveness  as  it  scales?  While  this   question  sounds  simple  enough  in  theory,  it  poses  several  methodological  and  practical  challenges.  This   paper  outlines  some  of  those  key  challenges  and  presents  potential  approaches  for  addressing  them.     Our  ongoing  work  examines  variation  in  impacts  across  KIPP  middle  schools.  To  some  extent,  many   “typical”  sources  of  variation  across  schools  aren’t  applicable  in  the  case  of  KIPP  due  to  its  consistent   *  http://www.ed.gov/news/press-­‐releases/nations-­‐boldest-­‐education-­‐reform-­‐plans-­‐receive-­‐federal-­‐ innovation-­‐grants-­‐once-­‐pr,  accessed  April  12,  2011.   SREE  Spring  2012  Conference  Abstract:  KIPP  i3  Evaluation,  Invited  Symposium   1 model.†  But  even  that  consistent  model  leaves  two  key  dimensions  of  potential  variation:  variation  in   impacts  across  the  different  grade  spans  of  KIPP  schools,  and  variation  in  impacts  between  previously-­‐ established  and  newly-­‐opened  KIPP  schools  as  the  network  grows  in  the  number  of  schools  and  students   it  serves.     First,  KIPP  is  intentionally  expanding  vertically—into  elementary  and  high  schools  in  cities  and  locations   where  KIPP  middle  schools  are  already  established.  Yet  the  evidence  on  KIPP  is  entirely  focused  on   middle  schools.  Will  the  positive  impacts  found  in  past  studies  of  KIPP  persist  to  the  same  extent  in   elementary  and  high  school?  There  are  reasons  to  believe  that  they  might  not.  At  the  elementary  school   level,  unlike  the  middle  school  level,  KIPP  schools  are  not  enrolling  kids  who  have  fallen  behind  in  school   for  five  or  six  years  already.  At  the  high  school    level,  the  issue  is  somewhat  different:  KIPP  is  educating   students  who  most  often  have  already  been  in  KIPP  (attended  a  KIPP  middle  school)  for  up  to  four  years.   In  this  way,  there  may  not  be  sufficient  “room”  to  generate  additional  impacts  for  these  students.     Second,  KIPP  is  also  expanding  “horizontally”—in  some  cases  into  new  cities,  but  primarily  by  adding   new  middle  schools  in  existing  markets.  In  this  way,  the  “treatment”  at  a  new  KIPP  school  may  not  be   the  same  at  newly  established  schools  as  at  the  early  (or  existing)  KIPP  schools,  since  it  may  be  harder  to   recruit  teachers  and  school  leaders,  attract  students,  and  so  forth.    On  the  other  hand,  schools  in   existing  markets  may  be  able  to  capitalize  on  economies  of  scale  from  regional  structures  and  supports   already  in  place.     Generally,  our  plans  for  estimating  impacts  include  a  combination  of  approaches.  We  will  evaluate   KIPP’s  impacts  on  student  achievement  by  capitalizing  on  the  advantages  of  both  experimental  and   quasi-­‐experimental  designs  (QEDs).  An  experimental  approach,  or  randomized  control  trial  (RCT),  can   provide  the  most  rigorous  assessment  of  impacts  on  student  outcomes,  but  it  may  be  applied  only  in   oversubscribed  schools  (those  with  more  applicants  than  available  slots)  that  hold  lotteries  to  determine   admission.  The  QED  may  be  less  rigorous  given  the  potential  for  selection  bias,  but  it  offers  the   advantage  of  applicability  to  most  KIPP  schools  in  that  oversubscription  is  not  requisite  for  comparison   groups  to  be  identified  from  similar  students  attending  nearby  schools.     To  complicate  matters  further,  neither  of  these  approaches  can  practically  be  applied  to  each  of  the   three  grade  spans  of  schools  (elementary,  middle,  and  high;  see  Figure  1).  As  it  currently  stands,   elementary  and  middle  school  entry  grades  both  represent  major  points  of  entry  into  the  KIPP  program,   and  oversubscription  at  those  grades  is  fairly  common.  To  date,  KIPP  high  schools  have  focused  on   serving  students  who  have  attended  KIPP  middle  schools,  resulting  in  a  pool  of  outside  applicants   potentially  participating  in  admissions  lotteries  too  small  to  support  an  RCT.  Conversely,  we  must   exclude  any  KIPP  elementary  schools  from  any  QED  approaches,  since  we  lack  pre-­‐test  data  for  their   students  (who  enroll  as  young  as  three  years  of  age  in  some  cases)—the  key  variable  for  generating  a   credible  comparison  group.     Given  these  considerations,  our  estimation  strategy  will  involve  three  different  types  of  designs:     1. RCT.  A  lottery-­‐based  RCT  will  compare  lottery  winners  (treatment  group)  to  lottery  losers   (control  group)  at  KIPP  elementary  and  middle  schools.   †  Nevertheless,  we  find  variation  in  impacts  across  different  KIPP  middle  schools,  ranging  from  -­‐0.12  to  +0.76   standard  deviation  units  in  math  and  -­‐0.14  to  +0.43  in  reading  after  one  year.  Potential  factors  related  to  this   variation  in  impacts  will  be  examined  in  a  forthcoming  report  expected  in  2012.   SREE  Spring  2012  Conference  Abstract:  KIPP  i3  Evaluation,  Invited  Symposium   2 2. Matched-­‐student  QED.  A  student-­‐level  propensity-­‐score-­‐matched  comparison  group  QED  will   compare  students  at  KIPP  middle  and  high  schools  (treatment  group)  to  similar  non–KIPP   students  (comparison  group).   3. Matched-­‐school  QED.  A  student-­‐level  QED  derived  from  school-­‐level  matching  will  compare   outcomes  for  KIPP  middle  school  students  with  and  without  the  opportunity  to  later  attend  a   KIPP  high  school  (comprising  the  treatment  and  comparison  groups,  respectively).  One  model   will  compare  students  within  a  region,  over  time  (before  and  after  the  KIPP  high  school  opens);   another  will  compare  a  single  cohort  of  students  across  regions  (with  and  without  KIPP  high   schools).     Using  these  models,  we  will  attempt  to  examine  three  main  sources  of  variation  in  the  impacts  of   KIPP  schools  on  student  achievement:     1. Are  newly  opened  KIPP  schools  in  areas  already  saturated  with  other  KIPP  schools  less  effective   than  the  previously  opened  schools?     2. Are  the  impacts  of  KIPP  schools  related  to  school  characteristics  that  may  be  associated  with  the   growth  of  the  network—such  as  key  characteristics  of  principals  or  students?       3. Is  the  KIPP  model,  shown  to  be  successful  at  the  middle  school  level,  as  successful  at  the   elementary  and  high  school  levels?     SREE  Spring  2012  Conference  Abstract:  KIPP  i3  Evaluation,  Invited  Symposium   3 Appendix  A.  References       Angrist,  J.  D.,  Dynarski  ,  S.  M.,  Kane,  T.  J.,  Pathak,  P.  and  Walters,  C.  (2010,  February).    Who  Benefits  from   KIPP?  National  Bureau  of  Economic  Research  (NBER)  Working  Paper  Series,  #15740.  Cambridge,   MA:  NBER.   Betts,  J.,  and  Tang,  Y.  E.  (2011,  October).  The  Effect  of  Charter  Schools  on  Student  Achievement:  A  Meta-­‐ Analysis  of  the  Literature.  Seattle,  WA:  National  Charter  School  Research  Project,  Center  on   Reinventing  Public  Education,  University  of  Washington.     Tuttle,  C.  C.,  Sullivan,  M.,  Gleason,  P.  M.,  Booker,  K.,  Furgeson,  J.,  Goble,  L.,  and  Knechtel,  V.  (2011,   October).  Evaluation  of  KIPP  i3:  Revised  Design  Report.  Washington,  DC:  Mathematica  Policy   Research.     Tuttle,  C.  C.,  Teh,  B.,  Nichols-­‐Barrer,  I.,  Gill,  B.  P.,  and  Gleason,  P.  M.  (2010,  June).  Student  Characteristics   and  Achievement  in  22  KIPP  Middle  Schools.  Washington,  DC:  Mathematica  Policy  Research.     Woodworth,  K.R.,  David,  J.L.,  Guha,  R.,  Wang,  H.,  &  Lopez-­‐Torkos,  A.  (2008).  San  Francisco  Bay  Area  KIPP   schools:  A  study  of  early  implementation  and  achievement.  Final  report.  Menlo  Park,  CA:  SRI   International. SREE  Spring  2012  Conference  Abstract:  KIPP  i3  Evaluation,  Invited  Symposium   A-­‐1 Appendix  B.  Tables  and  Figures       Figure  1.  Eligibility  of  Schools  for  Impact  Analysis       Elementary   Middle   High     RCT            (randomized   control  trial)     40%  of  schools   25%  of  schools   Lack  of  excess  demand   precludes  lottery-­‐based   analysis     QED            (quasi-­‐ experimental   Lack  of  pretest  prevents   80%  of  schools   60  %  of  schools   design)     matching  on  prior   achievement     SREE  Spring  2012  Conference  Abstract:  KIPP  i3  Evaluation,  Invited  Symposium   B-­‐1