An Evidence-based Adult Education Program Model Appropriate for Research John P. Comings and Lisa Soricone Harvard Graduate School of Education Maricel Santos San Francisco State University NCSALL Occasional Paper March 2006 Harvard Graduate School of Education 101 Nichols House, Appian Way Cambridge, MA 02138 NCSALL Occasional Papers are funded by the Educational Research and Development Centers program, Award Number R309B960002, as administered by the Institute of Education Sciences (formerly Office of Educational Research and Improvement), U.S. Department of Education. However, the contents of NCSALL Occasional Papers do not necessarily represent the positions or policies of the Institute of Education Sciences, or the U.S. Department of Education, and you should not assume endorsement by the Federal Government. CONTENTS TABLES..........................................................................................................................................iv ACKNOWLEDGMENTS..................................................................................................................v CHAPTER ONE: INTRODUCTION................................................................................................1 Four Program Components.........................................................................................................2 Relevance to Researchers..........................................................................................................3 Methodology................................................................................................................................4 The Next Step..............................................................................................................................5 Design of This Document............................................................................................................6 CHAPTER TWO: SUMMARY OF PRINCIPLES.............................................................................9 Program Quality Support.............................................................................................................9 Entering a Program.....................................................................................................................10 Participating in a Program..........................................................................................................11 Re-engagement..........................................................................................................................12 CHAPTER THREE: SUPPORTS TO PROGRAM QUALITY..........................................................13 Well-Defined Role.......................................................................................................................13 Effective Management System...................................................................................................18 Human Resource Management..................................................................................................28 Suitable Environment for Learning..............................................................................................43 CHAPTER FOUR: ENTERING A PROGRAM................................................................................51 CHAPTER FIVE: PARTICIPATING IN A PROGRAM....................................................................61 Classroom Management.............................................................................................................61 Instructional Process..................................................................................................................70 Supports to Participation.............................................................................................................93 CHAPTER SIX: RE-ENGAGEMENT............................................................................................109 CHAPTER SEVEN: RESEARCH AGENDA.................................................................................119 REFERENCES..............................................................................................................................121 APPENDIX: RESEARCH SITE IDENTIFICATION PROTOCOL (RSIP)......................................149 Overview...................................................................................................................................149 References...............................................................................................................................155 iii TABLES Table 1: Research Site Identification Protocol (RSIP)..................................................................156 Table 2: Research Site Identification Protocol (RSIP) Summary Sheet–for use by the reviewer........................................................................172 iv ACKNOWLEDGMENTS We would like to gratefully acknowledge the contributions of the professionals who reviewed earlier drafts of this monograph and provided substantive comments on it. Judy Alamprese, Abt Associates Kaye Beall, National Center for the Study of Adult Learning and Literacy Cristine Smith, National Center for the Study of Adult Learning and Literacy John Tibbetts, California Adult Literacy Professional Development Project American Institutes for Research Sally Waldron, World Education Heide Wrigley, LiteracyWork International In addition, we would like to thank the other individuals who offered constructive comments and reactions to this monograph: Erik Jacobson, American Institutes for Research Nancy Hansen, Sioux Falls Area Literacy Council, Inc. Tanya Tweeton, ESOL and GED programs, Broward County Schools, Florida v CHAPTER ONE: INTRODUCTION In 2003, the National Center for the Study of Adult Learning and Literacy (NCSALL) published Establishing an Evidenced-based Adult Education System,1 which described an approach to improving policy and practice. That monograph defined evidence-based practice as: The integration of professional wisdom with the best available empirical evidence in making decisions about how to deliver instruction.2 Empirical evidence is the knowledge researchers develop through well-designed, rigorous studies. Professional wisdom is the knowledge practitioners develop as they work with students, and as they take research findings and apply them in their programs. Neither source of knowledge alone is sufficient to ensure effective policies and practice, but together they provide the best guidance available for programs. Establishing an Evidenced-based Adult Education System describes steps that could lead to the establishment of an adult education3 system based on professional wisdom and empirical evidence. The first step is a review of the available empirical evidence and professional wisdom in order to define program models that meet the requirements for good practice. A program model describes what teachers, adult students, counselors, administrators, volunteers, and program partners (such as businesses, unions, health centers, and training programs) should do to provide both effective instruction and the support services adults need in order to persist in their learning long enough to be successful. The program model described in this monograph came about as the result of taking this first step. Although this model could also be used as a way to evaluate programs for other purposes, here it describes the context in which research on approaches to instruction and support services could be productive. Without well-defined program models, research may take place within programs that exhibit a range of quality, and some programs may have design elements that make research findings unreliable. For example, when research testing two approaches to teaching reading takes place within programs that do not have the support services necessary to help adults persist in their learning, the research may show no difference between the two approaches. This could be because the students were not engaged in learning long enough for a difference in the effect of the teaching approaches to be apparent. 1 This paper can be viewed at http://www.ncsall.net/fileadmin/resources/research/op_comings3.pdf 2 This definition can be viewed at http://www.ed.gov/offices/IES/speeches/evidencebase.html 3 In this monograph, adult education refers to adult basic education, adult English for speakers of other languages (ESOL), and adult secondary education programs that help students improve language, literacy, and math skills; acquire a high school equivalency; and transition to postsecondary education or training. 1 NCSALL Occasional Paper March 2006 Establishing an Evidenced-based Adult Education System also describes a range of methods that researchers could use to conduct evaluations of interventions meant to improve services, and suggests a hierarchy based on how well each method predicts impact.4 However, adult education programs and classrooms are complex enterprises designed for teaching and learning, not for the purpose of conducting research. Establishing an Evidence-based Adult Education System acknowledges that ethical, logistical, and budgetary constraints often make it difficult to employ rigorous methods. This monograph seeks to establish a way for researchers to choose programs that offer an opportunity to employ the most appropriate research method for identifying and evaluating effective interventions. Four Program Components This monograph describes program models as having a program quality support component, and three chronological program components: entrance into a program, participation in a program, and re-engagement in learning. The program quality support component is the organizational structure that supports • students and teachers in their efforts to learn and teach. This component includes the goals the program is attempting to achieve, the way it organizes that effort, the relevance of that effort, the resources needed to accomplish it, and the way those resources are improved over time. Entrance into a program includes recruitment—how programs describe and publicize • their services to attract students and help them make an informed choice to begin a course of study—and the intake and orientation process—how programs assess the needs and goals of students and prepare them to be successful learners. This component also includes wait-list management, as this issue addresses situations in which a learner wants to enroll in a program but cannot due to lack of available classes. Participation in a program includes instruction—how programs help students learn— • and support services—how programs help students participate, persist, and engage in learning long enough to reach their goals. Reengagement in learning includes the ways in which programs help students • continue learning after they stop participating in program services, resume participation when they are able, and begin postsecondary education or training after completion of program services. The four components that make up a program model are defined by principles. A principle describes a guiding assumption about how to design the program quality 4 For a debate about the prioritization of research methods, see Slavin (2002, 2004) and Belzer & St. Clair (2005) for divergent perspectives. 2 An Evidence-based Adult Education Program Model Appropriate for Research support component and three program components in ways that are sufficient for research. These principles are derived from empirical evidence and professional wisdom, and defined through a process that balances the advice from empirical evidence and professional wisdom with the constraints inherent in the field of adult education. Each program model should target a specific student population, which is defined by instructional goals and needs. Establishing an Evidenced-based Adult Education System describes several specific populations (such as beginning ESOL students who are literate in their own language, ABE students reading at the 5th to 8th grade level, or GED students with good literacy and numeracy skills). However, this paper describes a generic program model, without the specific elements for each population. Later publications will describe program models for each group of students, but those models will incorporate the principles described in this monograph. This monograph proposes a set of principles that describe programs in which researchers can conduct studies that compare different approaches to instructional and support services. Though this program model is based on the best available professional wisdom and empirical evidence, it is untested. This model should be tested to assess its impact on participation—in terms of gains in skills and knowledge—in programs that conform to it. Once the model has been tested, further research could evaluate the relative merits of different approaches to instruction and support, or evaluate different definitions of each principle. Chapter Seven sets out a research agenda to test and improve this model. Relevance to Researchers In his review of outcome and impact studies, Beder (1999) found that most studies were “flawed in ways that severely compromised the validity and utility of their findings” (p. 74). Even in those studies that were well designed and executed, the nature of adult education programs produced flaws that compromised the findings. Three specific aspects of research are dependent on program design, and the principles set out in this document address all three. Research that seeks to develop or evaluate interventions needs: A study sample of adult students who persist long enough in their programs to show • learning progress. Study samples of students who are at the same academic level (beginning, • intermediate, and advanced, for example) so that the findings can be generalized from the study group to other students who have the same needs and goals. Programs that have the ability to deliver services in accordance with the guidelines of • the proposed intervention. 3 NCSALL Occasional Paper March 2006 Programs that meet these three criteria for research are also programs that meet the needs of the students who are enrolled in them. Methodology The team that drafted this paper reviewed the existing adult education literature to discover a consensus about program design. After the review, the team concluded that: Only a few studies describe research that defines principles that could be included in • this paper. Several sources present theories that are based on research and observation or on • educational philosophy that define principles that could be included in this paper. The literature includes several attempts to describe the field’s professional wisdom in • terms of principles for good practice. These documents refer to each other and draw from the same sources, and, therefore, they represent an informed opinion drawn from the existing literature and the professional experience of well-respected practitioners and academics. Only some of the principles derived from the existing adult education literature are • defined in ways that would allow an observer to determine with confidence that a principle is demonstrated in a program. The team devised the following strategy to address the four limitations listed above. First, the team drew from the professional wisdom sources to define a set of principles. Then, the team looked at the theory and research literature to identify sources that support or contradict these principles. Finally, the team proposed ways to verify that a program is operating under these principles. After the first draft of this document was completed, it was shared with experts to seek feedback that informed this final list of principles. Since the research base in this field has few studies that test the impact of any of these principles, the input of this expert panel helped the team make the decision as to which principles are important, and how to verify that they exist within a program. A list of the experts consulted for this monograph can be found in the Acknowledgments. Although the team found a consensus on the set of principles, statements that defined how the principles could be identified in practice were usually unclear. For some principles, particularly those related to the program quality support component, the team was able to build on previous work in the field to propose a clear approach to verification. For other principles, particularly those related to instruction and support services, that clear approach seemed impossible without further empirical study. Readers should, therefore, view these principles as often representing one end of a continuum, not a specific point. 4