MEMORANDUM January 30, 2017 TO: Lance Menster Officer, Elementary Curriculum and Development FROM: Carla Stevens Assistant Superintendent, Research and Accountability SUBJECT: THE EFFECT OF THE WRITER’S WORKSHOP ON THE EXPOSITORY WRITING PERFORMANCE OF THE HOUSTON INDEPENDENT SCHOOL DISTRICT’S FOURTH-GRADE STUDENTS, 2015–2016 Writer’s workshop is a process approach to writing instruction using a workshop model being implemented in the Houston Independent School District’s kindergarten to fourth grade. The process involves prewriting or planning, drafting, revision, editing, and evaluation at students’ pace and, in some cases, the publication of writing pieces using a variety of genres, ideas, organizations, and tones. The purpose of this evaluation was to determine the effects of the Writer’s Workshop on teachers’ preparation for writing instruction, their actual writing instruction, and on students’ expository writing practices using teacher surveys, classroom observations, and treatment effects analysis of students’ writing performance. Key findings include: • According to 2015–2016 eTrain data, between 48.0 and 65.7 percent of teachers in three cohorts, who enrolled in the Writer’s Workshop professional development (PD), completed the training. • Fifty percent of 57 teachers who responded to the survey indicated that they dedicated 30 minutes to daily instructional writing. • With an average rating of 3.71 on a 5-point scale, teacher respondents agreed or somewhat agreed that the HISD curriculum resources were appropriate and effective for implementing Writer’s Workshop. • Six of the eleven Writer’s Workshop classrooms observed in April 2016 had 66.7 to 83.3 percent of the 24 components on the observation schedule based on one observer. • The treatment effect with regression adjustment analysis showed that fourth-grade students whose teachers completed the Writer’s Workshop professional development had an average raw score that was 0.22 items lower on the expository component of the 2016 STAAR writing test compared to the average raw score of students whose teacher did not complete the PD. The difference was statistically significant. Further distribution of this report is at your discretion. Should you have any further questions, please contact me at 713-556-6700. Attachment cc: Grenita Lathan Cindy Puryear H o u s t o n I n d e p e n d e n t S c h o o l D i s t r i c t RESEARCH Educational Program Report The effect of the writer's workshop on the expository writing performance of the Houston IndePendent school district's fourth-Grade students, 2015-2016 201 Board of Education 7 Wanda Adams President Diana Dávila First Vice President Jolanda Jones Second Vice President Rhonda Skillern-Jones Secretary Anne Sung Assistant Secretary Anna Eastman Manuel Rodriguez, Jr. Michael L. Lunceford Holly Maria Flynn Vilaseca Richard A. Carranza Superintendent of Schools Carla Stevens Assistant Superintendent Department of Research and Accountability Ted D. Serrant, Ph.D. Research Specialist Venia R. Holmes, DrPh Research Manager Houston Independent School District Hattie Mae White Educational Support Center 4400 West 18th StreetHouston, Texas 77092-8501 www.HoustonISD.org It is the policy of the Houston Independent School District not to discriminate on the basis of age, color, handicap or disability, ancestry, national origin, marital status, race, religion, sex, veteran status, political affi liation, sexual orientation, gender identity and/or gender expression in its educational or employment programs and activities. The Effect of the Writer’s Workshop on the Expository Writing Performance of the Houston Independent School District’s Fourth-Grade Students, 2015–2016 Executive Summary The Writer’s Workshop is an approach to writing instruction using a workshop model. It is being implemented in kindergarten to fourth-grade in the Houston Independent School District. The process involves prewriting or planning, drafting, revision, editing, and evaluation at the students’ pace and, in some cases, publishing of writing pieces using a variety of topics, genres, ideas, organizations, and tones. The Writer’s Workshop sought to improve students’ writing through teacher preparation by accommodating weaker students in research-based approaches that treat writing as a process rather than a product (Murray, November, 1972). Writer’s Workshop treats students as bona-fide authors, exposing them to the approaches authors use in their writing. Teachers were exposed to four days and/or 12 hours of training in the writing process in preparation for instructing students using Writer’s Workshop. Writing is seen as essential to effective communication, learning, and self-expression (Graham, Bollinger, Olson, D'Aoust, MacArthur, McCuthen & Longhouse, 2012). The purpose of this evaluation was to determine the effects of the Writer’s Workshop on teachers’ preparation for writing instruction, their writing instruction, and on students’ expository writing practices and performance. The effect of the workshop on students’ writing performance was determined using the State of Texas Assessments of Academic Readiness (STAAR) writing scale and raw scores and the percentage of students who met the state’s writing standards, a teacher survey, and classroom observations. Highlights  According to 2015–2016 eTrain data, between 48.0 and 65.7 percent of teachers in three cohorts, who enrolled in the Writer’s Workshop professional development (PD), completed the training.  Fifty percent of 56 teachers who responded to the survey question indicated that they dedicated 30 minutes to daily instructional writing.  The teachers who were surveyed had an overall rating of 1.70 on a 2-point scale with “1” being no and “2” being “yes” for their training and preparation for implementing Writer’s Workshop. With an average rating of 3.71 on a 5-point scale, teacher respondents agreed or somewhat agreed that the HISD curriculum resources were appropriate and effective for implementing Writer’s Workshop.  On a 5-point scale, teachers who were surveyed had an overall rating of 3.76 on a 5-point scale indicating that they agreed or somewhat agreed that they were efficacious and that they adhered to the writing instructional practices.  The teachers who were surveyed either somewhat agreed or agreed that their students, 3.49 on a 5- point scale, were able to practice writing using the fundamental Writer’s Workshop principles and components.  Six of the eleven Writer’s Workshop classrooms observed in April 2016 had 66.7 to 83.3 percent of the 24 components on the observation schedule based on one observer. WRITER’S WORKSHOP REPORT, 2015–2016  A comparable portion of fourth-grade students in the group whose teachers completed the Writer’s Workshop training (61.5%) and whose teachers did not complete the training (60.7%) met the Level II Satisfactory at the 2016 progression standard on the STAAR 2016 writing test.  Using a treatment effect with regression adjustment analysis, fourth-grade students whose teachers completed the Writers Workshop PD had an average scale score on the 2016 STAAR writing test that was 104 points lower than the scale score of students whose teachers did not complete the PD. The difference was statistically significant.  The treatment effect with regression adjustment analysis also showed that fourth-grade students whose teachers completed the Writer’s Workshop professional development had an average raw score that was 0.22 items lower on the expository component of the 2016 STAAR writing test compared to the average raw score of students whose teacher did not complete the PD. The difference was statistically significant. Recommendations  Teacher completion of the professional development designed to enhance preparation for effective delivery of the Writer’s Workshop needs to be encouraged. The high contract to participation cost incurred in delivering this professional development and the failure of teacher participants to confidently confirm students’ ability to outline the writing process using the Bomer’s Flowchart makes the case for the encouragement.  Teachers may require additional resources to effectively teach writing at the elementary school level. It may be essential to ensure that writing receives the same level of attention at all elementary grade levels in addition to the fourth grade where writing is tested.  Future evaluations should focus on observing the delivery of the Writer’s Workshop, particularly before the testing period, and possibly multiple times during the school year to assess teachers’ actual writing instructional practices. This would allow for a more reliable observation of the Writer’s Workshop and a more robust assessment of fidelity. HISD Research and Accountability____________________________________________________2 WRITER’S WORKSHOP REPORT, 2015–2016 Introduction The Houston Independent School District (HISD) implemented the Writer’s Workshop in kindergarten to eighth grades in all schools during the 2015–2016 school year. The Writer’s Workshop uses the process approach to writing instruction within a workshop model. The process involves prewriting or planning, drafting, revising, editing, evaluating at students’ pace, and in some cases the publication of written pieces using a variety of topics, genres, ideas, organization, and tones. Students are seen as bona-fide authors and, therefore, are exposed to the approaches authors use in their writing. In preparation for facilitating students’ writing, teachers of writing were exposed to a four-day and/or 12-hour training in the writing process or Writer’s Workshop. Writing has been described as an essential aspect of “professional, social, community, and civic” engagement because the ability to write well is fundamental to effective “communication, learning, and self- expression” (Graham, et al., 2012, p. 6). A national survey of elementary school teaching instruction, grades 4 – 6, found that teachers were inadequately prepared to teach writing, students’ time devoted to writing was inadequate, and that students’ writings were commonly restricted to narratives, letter writing, completing worksheets, and responding to materials read (Gilbert & Graham, 2010). There was little evidence that teachers used research-based approaches when teaching writing, and that they made few or no adaptations or did so infrequently to facilitate weaker students (Gilbert & Graham, 2010). On the 2007 National Assessment of Educational Progress (NAEP) writing assessment, 67 percent of eighth-grade students performed at or below the basic level in writing indicating that they had attained “only partial mastery of the writing skills needed” at this grade level (National Assessment of Educational Progress, Salahu-Din, Persky & Miller, 2008 cited in Gilbert & Graham, 2010). Sixty-two and 63 percent of HISD fourth- and seventh-grade students, respectively met phase-in 1 Level II: Satisfactory standard on the 2015 State of Texas Assessments of Academic Readiness (STAAR) writing test, compared to 70 and 72 percent respectively at the state level. STAAR writing in HISD and the State of Texas is tested in the fourth and seventh grades. The Writer’s Workshop seeks to improve students’ writing through teacher preparation, use of a variety of genres and topics, and use of research-based approaches that treat writing as a “process rather than a product” (Murray, 1972, p. 3). The purpose of this evaluation, therefore, is to determine the effects of the writing workshop on teacher preparation for teaching writing, writing instruction, and students’ expository writing practices and performance. The effect of the workshop will be determined using STAAR writing scores and performances and Stata treatment effects with regression adjustments (teffects ra), a teachers’ survey, and classroom observations. The evaluation sought to answer the following four questions: 1. What were the perceptions and experiences of teachers who participated in the Writer’s Workshop? 2. What were the demographic characteristics of the Writer’s Workshop fourth-grade student sample? 3. How did fourth-grade students whose teachers participated in the Writer’s Workshop perform on the overall and expository component of the 2016 STAAR writing assessment? 4. What was the effect of the Writer’s Workshop instruction on the overall and expository writing performance of fourth-grade students whose teachers completed the Writer’s Workshop? HISD Research and Accountability____________________________________________________3 WRITER’S WORKSHOP REPORT, 2015–2016 Literature Review Expository writing involves communicating information about a topic using facts and details. It includes specific terms and materials that help classify and define the topic. This type of writing is organized using structures such as examples, classification, cause and effect, problem and solution, and compare and contrast (Houston Independent School District, 2015; Purdue University, 2016). The narrative and expository writing samples of sixth- to twelfth-grade students with language learning disabilities were evaluated for elements of microstructure1 and macrostructure2 using experimentation. Results of the Wilcoxon signed ranks tests indicated that the narrative genre had significantly higher levels of productivity and grammatical complexity compared to the expository genres. Both genres showed impoverished text structure and revealed strong correlations between the microstructure and macrostructure (Hall-Mills & Apel, 2013). A report on a 2008 national writing survey concluded that despite consensus on research-based approaches to writing, they have not been widely implemented (Laud, 2013). In addition, most schools have insufficient comprehensive, sustained, and focused systems for offering teacher professional development to support research-based writing practices (Laud, 2013). Research on the effects of the Writer’s Workshop has been mixed. A study involving 90 kindergarten students, randomly assigned to journal writing and Writer’s Workshop classrooms, revealed a significant difference in students’ writing achievement in favor of those who were exposed to the Writer’s Workshop strategies using a repeated measures test on a 10-stage developmental writing rubric (Mester, 2011). When a Chi-square contingency table was used to determine the effect of the Writer’s Workshop on 321 students in achievement on the Georgia fifth grade writing assessment over three years, the results showed that students’ initial performance prior to the intervention was the highest (Smithson, 2008). In addition, independent samples t-test of the latter two years’ performance revealed no significant scale score difference between the groups’ writing performance on the assessment (Smithson, 2008). Smithson (2008) identified at least three factors that may explain the unexpected findings of the Georgia study: (1) teacher resistance to full implementation of Writers Workshop due in part to inadequate training and teacher preparation, (2) changes in the test ratings and achievement standards, and (3) lower student ability over time. The writing performance of 18 first-grade students was compared using writing prompts and Writer’s Workshop strategies during a five-week period. The result showed that the reading attitudes of both groups declined while the group exposed to the prompts demonstrated a larger decline. Students who were taught using prompts, however, had mean scores that were higher than those who used the Writer’s Workshop strategies. Overall, students who were better writers enjoyed the writing process more (Caroll & Feng, 2010). This study, however, did not specifically investigate Writer’s Workshop’s effect on expository writing. The United States Department of Education Institute of Education Sciences (IES) outlined four recommendations for teaching elementary school students to be effective writers (Graham, Bolinger, Olson, D’Aoust, MacArthur, McCuthen & Longhouse, 2012). These were (1) provide daily time for students to write; (2) teach students to use the writing process and how to write for a variety of purposes; (3) teach students to become fluent with handwriting, spelling, sentence construction, typing, and word processing; and (4) create an engaged community of writers. (Graham, et al., 2012). Writer’s Workshop, as an approach to the writing process, satisfies the Institute’s recommendations for teaching elementary students to be effective writers. Writer’s Workshop’s effectiveness is the focus of this evaluation study. 1 Microstructure involves the text base, writer’s conveyance of meaning at multiple level (word, sentence), productivity, grammatical complexity and accuracy, and lexical diversity 2 These include genre-specific text structure elements like the “gist of the text, organization, cohesion, and text structure, writer’s conveyance of meaning. HISD Research and Accountability____________________________________________________4 WRITER’S WORKSHOP REPORT, 2015–2016 Methods Data Collection and Analysis Data collection for the evaluation of the Writer’s Workshop included an online survey, classroom observations, and analysis of students’ overall and expository writing results on the 2016 STAAR fourth- grade writing assessment. STAAR writing in elementary school is tested only in the fourth grade. The evaluation benefited from several meetings and discussions with the elementary curriculum officer and managers responsible for supervising the implementation of the Writer’s Workshop including the professional development (PD) component. It also involved the review of the PD documents and other online document resources related to writing and the Writer’s Workshop. An online questionnaire was administered to teachers who completed the Writer’s Workshop PD training. The survey questions were developed based on the course content. The questionnaire was vetted for validity by the elementary curriculum officer and managers, which they disseminated to increase teachers’ response rate. The survey was administered using SurveyMonkeyTM. SurveyMonkeyTM is a free web-based software that administers, collates, and analyzes survey data. The survey was administered from May 12 through 27, 2016. SurveyMonkeyTM calculated an average rating scale score (See Appendix A on p. 19 for details). The survey results constitute a major part of this research report. A sample of schools were randomly selected for classroom observations based on their performance on the 2014–2015 STAAR writing test. Schools were ranked according to the percent of students who met Level II: Phase-in 1 Satisfactory standard and stratified into deciles. One school from each decile stratum was selected by random resulting in a total of ten schools. Emails were sent to the principals of these schools explaining the purpose and procedures for the visit. Six schools agreed to the visit. Classroom observations were conducted during regular fourth-grade writing instruction classes. Teachers were informed of the visit beforehand. All visits were scheduled between April 11 and 22, 2016 based on the English language arts (ELA) Pacing Calendar3 to coincide closely with expository writing instruction and to accommodate full implementation of the Writer’s Workshop and STAAR testing. However, two of these visits had to be rescheduled and were held April 27 and May 13, 2016. An observation schedule was developed using information from the “Writer’s Workshop Facilitator’s Guide” and “The First 25 Days of Writer’s Workshop” implementation document. Principals or their representatives were debriefed following each school visit to discuss observations and possible recommendations for improvements in program implementation. The results of the school visits were used to determine the program’s apparent4 fidelity. Fidelity has been described as “the proportion of a program’s components that was implemented” (Mowbray, Holter, Teague & Deborah, 2003, p. 316). Teachers serving students from kindergarten through fifth grade and who attended the summer Writer’s Workshop training in June and July 2015 were included in this study. They were selected because their students, particularly in the fourth grade, would have had a full year’s exposure to the teacher’s writing instruction skills and expertise acquired during the Writer’s Workshop PD training. Other training sessions were held in September, October, and November 2015, and January 2016 for kindergarten to second grade and third to fifth-grade teacher cohorts. 3 The ELA Pacing Calendar is a general guide prepared by HISD Curriculum and Development Department for the scheduled delivery of key Texas Essential Knowledge and Skills (TEKS) and ELA curriculum components. 4 I described it as apparent because I was unable to actually observe Writer’s Workshop lesson delivery in a number of cases since observation occurred post-assessment when active delivery of the writing process had ended. HISD Research and Accountability____________________________________________________5 WRITER’S WORKSHOP REPORT, 2015–2016 Participation in the Writer’s Workshop PD training was voluntary or principals could have recommended teachers. Only teachers who registered for the PD, however, were included in this study. This study, therefore, is restricted to teachers who either completed or were recorded as “no show” in the Writer’s Workshop PD sessions. Teachers and their students who completed the PD comprised the treatment group and students and their teachers who were “no show” made up the non-treatment group. Students’ raw scores (number of items correct) on the 2016 STAAR fourth-grade expository writing component and their overall writing scale scores were used to analyze students’ performance and determine program effect. Students’ overall writing scale scores and performance data were also used to determine the mean scale score difference between students whose teachers completed the program and teachers who were recorded as “no show” in the program. Analyses were based on the number of fourth-grade students who met the 2016 STAAR Level II: Satisfactory and Level III: Advanced writing performance at the 2016 progression standard. Sample The sample consisted of students whose teachers completed the Writer’s Workshop PD training and those whose teachers were recorded as ‘no show’ and whose students scored on the fourth-grade STAAR writing test. A total of 892 teachers registered to participate in the professional development program. These participants and their writing students constituted the “frame” from which the fourth-grade student sample was selected (see Murnane & Willet, 2011, p. 47). Generalization of results are limited, therefore, to the broader group of fourth-grade students whose teachers registered to participate in the program and not to the total population of fourth-grade students (see Murnane & Willet, 2011, p. 47). Forty-three fourth-grade teachers completed the PD and 12 fourth-grade teachers were recorded as “no-show”, making up the fourth- grade teacher sample. Students were linked to their fourth-grade teachers using the Public Education Information Management System (PEIMS) and the STAAR assessment result files. These files included demographic and test data for each student. A total of 2,371 fourth-grade students comprised the sample. Of these, 1,879 students had teachers who completed the Writer’s Workshop training and 492 students had teachers who were recorded as “no show” in the program. Tests of normality using detrended normal Q-Q and normal Q-Q plot of the scale score and the raw score for the composition (expository) component of the STAAR test were confirmed. Of the 447 teachers who completed the PD, 57 (12.8%) completed the survey questionnaire. As mentioned earlier, 11 writing classes in six of the 10 randomly selected schools were observed. A rubric of Writer’s Workshop criteria was used to conduct the evaluation by one observer (See Appendix B, p. 20). Data Limitations  As noted, the STAAR writing assessment is limited to fourth grade and, therefore, does not cover the full spectrum of grades whose teachers participated in the Writer’s Workshop.  Because of the limited survey administration timeframe, only 12.8 percent of teachers responded to the Writer’s Workshop survey. Any inference about the survey has to be restricted to that group of teachers since they may not be representative of the entire population of Writers Workshop participants in HISD.  Classroom observations were conducted toward the end of the school year to minimize disruption. Instruction had to be scheduled to accommodate the visits and may not have reflected the preferred natural setting for observation. HISD Research and Accountability____________________________________________________6