Forum on Public Policy Overcoming Educational Challenges to Women Living in At-Risk Communities through Urban Debate Carol K. Winkler, Associate Dean, Humanities, Georgia State University C. Kevin Fortner, Assistant Professor, Educational Policy Studies, Georgia State University Sara Baugh-Harris, Communication Studies Ph.D. Candidate, Georgia State University Abstract Every year 1.3 million U.S. high school students drop out of school with one quarter of female students failing to graduate on time. Female dropouts are more likely to be unemployed, to earn less when they are employed, to become pregnant before the age of 20, to become obese, to smoke, and to drink more heavily than their male counterparts. This study focuses on two public school systems with high rates of dropouts to determine whether urban debate leagues (UDLs) improve the affective, behavioral, and cognitive dimensions of school engagement of middle and high school females. The methods used in the analysis include: (1) propensity score matching to compare behavioral and cognitive indicators of debaters vs. non-debaters, (2) a nationally normed pre/post reading test, and (3) an alumni survey of UDL participants. The study found that debaters were significantly less likely to be tardy from school, scored significantly higher on standardized reading exams, and substantially exceeded national norms for annual progress in reading rate, accuracy, fluency, and comprehension. Program alumni reported that UDL participation increased both the skills and confidence levels needed for success in college and their careers. Introduction Approximately 1.3 million students in the United States drop out of high school each year (Office of the Press Secretary, 2010). More than half of those failing to graduate on time attend schools in the primary school systems of the nation’s 50 largest cities (Swanson, 2009). Schools in urban districts are particularly at risk for producing high numbers of dropouts because they serve large concentrations of students who share documented predictive indicators for dropping out of school: those who are African-American or Hispanic minorities (Maxwell, 2012; Newcomb et al., 2002; Prater, Sileo, & Black, 2000; Snyder & Sickmond, 2006, Swanson, 2008), those who belong to families with low incomes (Newcomb et al., 2002), and those who have comparatively lower levels of academic achievement (Battin-Pearson et al., 2000; Finn & Rock, 1997; Krohn, Thornberry, Collins-Hall, & Lizotte, 1995; Storm and Boster, 2007). In an effort to improve urban students’ academic engagement and timely progression to graduation, 26 U.S. cities have implemented Urban Debate Leagues (UDLs). Begun in Atlanta, Georgia in the 1980s, UDLs have historically served more than 40,000 students and are now active in more than 500 schools nationwide (NAUDL, 2012). Previous research on the leagues’ impact has been promising. One study reported that African-American males who participated 1 Forum on Public Policy in the Chicago UDL from 1997 to 2006 were 70% more likely to graduate, 3 times less likely to dropout, and more likely to meet ACT benchmarks for college readiness in English and Reading than a matched sample of their non-debate peers (Mezuk, 2009). Another study of the same city’s UDL focused on poor students with low grades who held special education status. It found that those debaters were 3.1 times more likely to graduate and more likely to reach college-ready benchmarks for English, Reading and Science on the ACT than their non-debate peers (Anderson & Mezuk, 2012). The remaining published studies attempting to quantify the impact of UDLs have focused on the totality of their league populations without consideration of how different genders respond to such programs. Studies combining the male and female participants have found that debaters were more likely to graduate, meet college readiness benchmarks, and improve their cumulative GPAs than their non-debate peers (Mezuk et al., 2011); debaters improved their reading levels beyond national norms for annual progress (Winkler, 2010); and debaters substantially improved their school attendance and conduct after only one year of participation in the activity (Winkler, 2011). No published study to date has focused on the impact of UDLs on female participants. The oversight is troubling for only one in four U.S. females completes high school on time. The problem worsens for minority females. Only 37 percent of Hispanic women, 40 percent of African American women, and 50 percent of Native American women graduate from high school on time (National Women’s Law Center, 2007). The need to focus on female participants becomes even more compelling when the consequences of leaving school receive consideration. Of all U.S. students who dropped out of high school in 2006, for example, males were more likely to find employment than females by a margin of 77 to 53 percent. Female dropouts on average earned $6000 less per year than females who stayed in school, a figure representing only 63 percent of the wages earned by male dropouts during the same period. When compared against females who stayed in school, female dropouts were also more likely to become pregnant before the age of 20, to become obese, to smoke, and to drink more heavily (National Women's Law Center, 2007). A key reason why females held back a grade in school were twice as likely to drop out as their male peers has been that female students, in particular, viewed their lack of timely progression as embarrassing (Fine & Zane, 1991). This study will help fill the research gap by examining how UDLs impact the school engagement of female participants in two urban school districts. School engagement is a multifaceted concept that encompasses affective, behavioral, and cognitive dimensions (Fredricks, Blumenfeld, & Paris, 2004). Accordingly, the study focuses on four research questions: RQ1. Does participation in UDLs improve the behavioral engagement of females in middle and high school? RQ2. Does participation in UDLs improve the cognitive engagement of females in middle and high school? 2 Forum on Public Policy RQ3. Does participation in UDLs improve the cognitive engagement of high school females who have fallen below grade level for reading? RQ4. How do female UDL alumni evaluate the lasting impact of UDL programs on their own cognitive, behavioral, and affective dimensions of school engagement? To address these questions, the following will describe the UDL program, explicate the study's methodology and results, and discuss the research findings. The Urban Debate League Program Prior to the start of the school year, each league begins by hosting a one-week, full-day summer debate workshop staffed by high school coaches, college debate coaches, UDL alumni, and intercollegiate college debaters. The workshop introduces the attendees to the fundamentals of debate: argument construction, evidence evaluation, organizational skills, efficient note taking, and oral presentation skills. It also familiarizes the students with the annual national debate topic. Examples include the 2011-12 topic, "Resolved: The United States federal government should substantially increase its exploration and/or development of space beyond the Earth's mesosphere" and the 2012-13 topic, "Resolved: The United States federal government should substantially increase its transportation infrastructure investment in the United States." Most importantly, the summer workshops frequently employ game formats as teaching methods to underscore that debate can be fun. During the academic school year, participating schools offer 2-4 hours of classroom or after-school debate instruction per week. During these sessions students practice skill building in debate fundamentals (described above), anticipate their future opponents' argumentative strategies, learn the vocabulary and arguments in their pre-packaged argument/evidence packets, review judge’s ballots from previous competitions to focus on needed areas of improvement, and participate in spar debates, rebuttal re-works, and other abbreviated forms of debate. Throughout the academic year, each league hosts 8 debate tournaments where students participate in switch-side debating to alternatively affirm or negate a proposition during 3-4 rounds of inter-scholastic competition. High school debate coaches, college debaters, and community members serve as judges after receiving training designed to encourage constructive suggestions for individual or team improvement. Students compete in novice, junior varsity, or varsity division based on their previous experience and success with debate. Students receive individual and team awards at the competitions’ conclusions based on their performances in tournament rounds of competition. The philosophy of the UDL is to present numerous awards in an effort to maximize the number of students who have positive, successful experiences in the academic competitions. Student recruitment into the UDLs occurs through a combination of methods. School- wide assemblies, teacher-led targeted recruitment of students deemed unmotivated by school, 3 Forum on Public Policy peer-led recruitment of friends, intercom/poster announcements of upcoming debate meetings, and after-school snacks are all strategies implemented to encourage student participation. In many urban debate leagues, summer workshops occur on college campuses, which can also serve to attract certain students. UDLs deploy various media formats, such as Twitter, Facebook, and movies about debate such as The Great Debater starring Denzel Washington, to help attract new members into the UDL community. Many school principals display the students’ accomplishments in trophy cases and publicize those successes in school-wide announcements. Student retention in UDL programs is a multi-faceted process. Initially, each of the participating leagues provides pre-packaged evidence and argument packets in order to reduce the stress about the workload associated with debate and to "level the playing field" by providing equal access to debate resources. The program's pre-packaged materials address aspects of the national topic deemed of interest to the students (e.g., a focus on space colonies during the space topic and a focus on social exclusion due to inadequate transportation for poor, minority families during the transportation topic). Novice workshops and school debate meetings rely heavily on a game format to teach debate fundamentals (e.g., raps filled with debate terminology, speed and articulation drills, spar debates on topics where the students are already knowledgeable about the content, etc.). Having variable divisions based on ability level in tournament competition reduces the chances that more-experienced peers will overwhelm beginning students to the point that they leave the program. Once participants overcome their initial fears about debate, they respond to different retention strategies. The primary one is competition. Just as competitive sports have ongoing attraction for many participants, debate is an academic game with winners and losers that builds skills during the process. Further, debaters develop friendly rivalries with competitors from other schools, driving them to remain in the activity to defeat their opponents in future competitions. Moreover, UDLs offer a stable community that students can rely upon regardless of whether their home address or school changes. Other retention strategies for more advanced high school participants include invitations to serve as peer mentors or judges in middle school competitions. Frequently, UDL students have active and inactive periods in tournament participation based on challenging situations at home or school. Rather than remove them from the program, league directors encourage the students to return whenever and as soon as they can. Method Participants To examine what effect, if any, UDLs had on the cognitive and behavioral engagement of female participants (RQ1; RQ2), this study examined the programs’ effects in two of the nation’s principle school districts. Both UDLs were serving school systems falling in the ten lowest performing urban districts nationwide for moving students through to graduation in a timely manner. Together, the two school districts were serving more than a million students, but neither 4 Forum on Public Policy district moved more than 45 percent of their students to graduation within four years (Swanson, 2009). The study examined the effect of UDL programs on middle school debaters in one school district and high school debaters in the other. The need to bifurcate the district sample by school level resulted from the implementation of the UDL programs. During the one year time frame used to analyze league data related to RQ1 and RQ2 (June 2011-June 2012), one city's school district provided high school programming exclusively, while the other devoted the vast majority of its resources to serving middle school students. Debate participants were students in grades 6 through 10 who attended at least one interscholastic debate competition during the academic year. Together, the middle and high school UDL programs served 245 students (203 in middle school and 42 in high school). Among the middle school students, 117 had complete administrative records for the year prior to their participation in debate. Twenty-five of the high school program students had complete administrative records for their 8th grade year. These limitations left about 58 percent of the debate participating students in our models for analysis. Table 1 (MSD) provides descriptive statistics on the 2010-11 school year characteristics of the middle school debaters and their matched comparison group of non-debaters. The final two columns of Table 1 present statistics comparing the means of the non-debate participant students to the overall school population and the reduction (or increase) in the difference as a result of our study's creation of matched comparison groups respectively. For example, the 0.079 value for Male indicates that the overall school population contained about 7.9 percent more males than the debate participating student sample. After matching, this difference in means was reduced by 35 percent. As Table 1 demonstrates, the middle school students participating in the league were not representative of their school-wide populations. In the middle school participant sample, tournament debaters were less likely to be of black ethnicity, to have participated in limited English proficiency programs (LEP), and to be eligible for free lunch than their school-wide population of peers. The UDL participants also had generally lower rates of absences for excused and unexcused reasons and had lower rates of disciplinary incidents. Test scores for students participating in debate were dramatically higher than the comparison pool students (about one- half to seven tenths of a standard deviation higher compared to the average student). After matching, the overall observable differences between the groups were substantially reduced, with the average test score difference falling by about 90 percent. In three cases, however, our matching procedure increased the differences between the two groups: underage for grade, homeless, and times tardy in 2010-11. Table 2 provides descriptive statistics for high school students in the comparison pool, debaters, and the study's matched sample. The group comparisons columns indicate differences between the overall school-wide population of students (8th Grade Comparison Pool), tournament debate participants, and the changes in those differences after the study's matching procedure. 5 Forum on Public Policy The participants in the high school UDL program were also not representative of the school-wide populations of the participating school districts. Females made up a larger percentage of the league’s participants in the analysis sample, a figure about 10 percentage points higher than in the school-wide population. The ethnicity of the two groups was also different, with African Americans making up a larger percentage of the league participants compared to the school-wide population. Further, Hispanics constituted 22 percent of the school-wide population, but none participated in the UDL. The pool of comparison students had higher rates of free lunch eligibility, but lower rates of reduced priced lunch eligibility. Tournament debaters had lower rates of both excused and unexcused absences, but slightly more disciplinary incidents. Finally, the high school debate tournament participants scored higher than their school-wide peers across all 8th grade standardized test scores in reading, math, English/language arts, science, and social studies. While the study's matching procedure increased the differences between groups for the other ethnicity group, days absent 2010-11, times tardy 2010-11, and the number of disciplinary incidents, all other observed differences between debate participants and other peers in the school were reduced. The difference in prior test score performance was reduced by about 70% across the five standardized tests used to match students. To examine what, if any, effect UDLs had on the cognitive engagement of female high school participants who have fallen below grade level for reading (RQ3), the study focused on participants who attend at least one debate tournament from 2006-07 to 2010-11 during grades 9- 12. All of the participating students were drawn from the school district described above that devoted all its programming resources to the high school level. As participation in this component of the study required signed parental consent forms, signed student assent forms, and time availability to complete both the pre- and post-test conditions of the measuring instrument, the sample size was 132 students, 17 percent of the total UDL population in that city over the years 2006-07 through 2010-11. Of those, between 25 and 34 students fell below grade level for reading prior to their involvement in debate based on which aspect of reading proficiency was tested. To examine what, if any, effects UDLs had on alumni’s affective, behavioral, or cognitive dimensions of school engagement (RQ4), the study utilized a convenience sample drawn from UDL programs nationwide. League directors throughout the United States received a message posted on the national UDL list serve asking for their help in forwarding an attached invitation to participate in the study to any female alumni who they could still contact. Twenty- three women from Atlanta, GA, Baltimore, MD, Boston, MA, Milwaukee, WI, Minneapolis/St. Paul, MI, Newark, NJ, New York, NY, and Oakland, CA responded to the survey. Measures To determine if UDL programs increased the behavioral and cognitive engagement of participating females (RQ1; RQ2) and to reduce any selection bias resulting from the voluntary 6 Forum on Public Policy nature of UDL participation, the study compared UDL debaters with groups of their non-debate peers based on propensity score matching (Rubin, 1973). Propensity score matching (Rosenbaum and Rubin, 1983; Rosenbaum and Rubin, 1985) modeled the probability of selection into a treatment based on observable pre-treatment characteristics. The resulting information was then used to construct a comparison group whose likelihood of treatment was similar to the treatment group. For this analysis, we coded students who participated in debate tournaments as the treatment condition and all other students in the same schools were available for participation in the matched comparison group. We created the comparison groups in Stata 12 using the psmatch2 algorithm (Leuven and Sianesi, 2003) with nearest neighbor matching as recommended by Henry and Yi (2009). The regression models analyzed the difference in outcomes for debate participation students and students in the matched comparison group based on propensity score matching. In an effort to find students similarly likely to participate in debate, our matched comparison groups were based on demographic factors (e.g., gender, ethnicity, socio-economic status), family-related factors (e.g., mother at home, father at home, number of siblings), academic risk factors (e.g., special education status, English Language Learner status, and over age for grade), behavioral engagement indicators (i.e. number of absences, excused absences, days tardy at school, and suspensions), and academic factors (e.g. GPA quality points, credit values of courses taken, and standardized test scores). For the groups matching the study's middle school cohort, we extracted information from the student’s records during their prior year of schooling (5th -7th); for the high school cohort, we used data from the students’ 7th and 8th grade years in school to identify the comparison group. The exact variables differed slightly as different school systems collect different student characteristics as part of their administrative data collection systems. This study combined the middle and high school data to create a common dataset for analysis of the outcomes associated with debate participation. To assess whether UDLs increased participants’ behavioral engagement (RQ1), our outcome indicators were the number of excused absences from school, the number of unexcused absences from school, the number of tardy incidents, and the number of disciplinary incidents. To assess whether UDLs increased participants’ cognitive engagement (RQ2), our outcome indicators included standardized test scores on end of grade exams in reading and mathematics. Because the exams occurred across systems and grade levels, scores were standardized within grade and test to generate a standardized test score. Using the entire universe of test takers in the district in a given year, we retained the single highest test score for each student after excluding test score values that were outside the range of plausible scores. Next, we standardized the test scores to have a mean of zero and a standard deviation of one. Our models using test score outcomes as the dependent variable utilized OLS regression with robust standard errors. Due to the severe non-normality of outcome distributions for models with count data such as disciplinary incidents and absences, we utilized negative 7 Forum on Public Policy binomial regression models with robust standard errors (Long and Freese, 2006). The models included the propensity score as a regression control and a variety of other control variables unrelated to debate participation that may have influenced the outcomes for students in the year of debate participation. The following provides the reduced form equation used for the test score related regressions. where is student i’s test score achievement in the current year, is the y-intercept, is - student achievement in a prior time period, is a vector of individual student characteristics both time varying and time invariant, is an indicator variable indicating a student’s participation in the urban debate league program, and the individual error associated with the observation of student i at time t. The models testing for gender specific effects introduced an interaction between the Debate indicator and female indicator where observations that are both female and debate participants equal 1 (the product of the Debate and female variables). To determine if UDL participation increased the cognitive engagement of high school females who fell below their grade level in reading, we used an additional measure of student reading to provide information (RQ3). The need for the supplementary instrument arose because of the relevant school districts' decision to last administer its standardized tests during their students’ 10th grade year(s) in school. To ensure the inclusion of results from 11th and 12th grade participants, we administered Gray Oral Reading Tests (GORTs), a reliable and valid national reading test which evaluates student gains in reading rate, accuracy, fluency, and comprehension against national norms for annual progress (Wiederholt & Bryant, 2001). In both pre- and post-test conditions, each student read passages aloud for 30 to 45 minutes to trained GORT administrators. The results were then compared against charted norms for reading that spanned from the 1st to the 12th grade. For the purposes of this study, students whose pre-tests showed that they had less than a year’s growth potential on rate, accuracy, fluency, and/or comprehension were removed from the calculations of the league-wide averages for annual reading progress, as such scores would artificially deflate any findings of reading progress. To discern how female UDL alumni evaluated the lasting impact of the program on the three dimensions of school engagement (RQ4), we administered an online survey. The survey began with questions related to the respondents’ age, ethnicity, number of years of UDL participation, city of UDL participation, highest grade level achieved, and current occupation. Open-ended questions followed that were related to academic success (level of preparation, ability to achieve goals, skill development related to goals, management of challenges, class participation, accessing college opportunities, and the use of college mentors), interpersonal interactions (with opposite and same-sex peers, instructors, and family members), and community engagement (involvement in community activities, management of community challenges, ongoing connections with UDL mentors, and on-going connections with UDL peers). 8 Forum on Public Policy Results Propensity Score Matching Results for RQ1 and RQ2. Table 3 presents the results of regression models comparing the outcomes of students on four measures of behavioral engagement and two measures of cognitive engagement. Each outcome model is presented twice. The first model result of each outcome pair estimates the difference in outcomes for students participating in debate tournaments to the matched comparison group. The second model includes the interaction between the dichotomous female variable and the debate treatment indicator variable. This interaction term estimates the difference in treatment effects by gender. A statistically significant coefficient indicates that the treatment effect for girls is statistically different from the treatment effect for boys. The resulting models indicate that debate tournament participation is equally effective for boys and girls in terms of influencing the treated student outcomes. Debate participation was associated with a reduced number of times tardy during the year. The coefficient (-0.718) represents the reduction in the log of the expected number of times tardy for students participating in debate compared to non-participants holding the other variables constant. Expressing this difference as an incidence rate ratio, debate participants are expected to decrease their rate for tardy attendance incidents by a factor of 0.488 compared to similar non-debate students. The models comparing the association between the other behavioral outcomes and debate participation revealed no statistically significant relationships between the treatment and the number of excused absences, the number of unexcused absences, or the number of disciplinary incidents. The p-value for the relationship between debate participation and the number of unexcused absences fell just outside our cut-off for statistical significance (0.054). Students participating in debate scored, on average, about 2/10ths of a standard deviation higher (0.221) than comparable students on standardized reading tests holding other characteristics constant. On standardized math tests, the students participating in debate and the comparison group students scored similarly when examining the results of a model that did not differentiate by gender. In the second standardized math outcome model, the coefficients differ sharply for male and female debate participating students with a -0.176 coefficient for male debate tournament participants and a 0.291 coefficient for female debate tournament participants. The t-statistic for the female debate interaction term has a p-value of 0.073, which does not meet the 0.05 cut-off for statistical significance. The introduction of these interaction effects did inflate the standard errors due to the decreased cell size of the groups. Gray Oral Reading Test Results (RQ3). The GORT results demonstrated that female high school students who entered the UDL reading below grade level exceeded national norms for annual progress on all four dimensions of reading proficiency. On the dimension of reading rate, forty-eight percent (n = 31) of the females who began the program read below grade level with an average pre-test score falling more than two years behind national norms for their grade level (8.0 vs. 10.3). Their average post-test score 12 months later showed rate progress of almost 9 Forum on Public Policy two years (1.9). On accuracy, 42 percent (n=34) entered the UDL reading below grade level, averaging more than two and half years below national norms for grade level on the pre-test (7.7 vs. 10.3). Twelve months later the female participants had improved their reading accuracy, on average, by a year and a half (1.5). On fluency, 50 percent (n=29) began the program reading below grade level by more than two and a half years for their grade level (7.6 vs. 10.2). After 12 months of UDL programs, they had improved their fluency by a full 2 years when compared against national norms. Finally, on comprehension, 43 percent (n-25) began the program reading below grade level by two and a half years (7.7 vs. 10.2). With twelve months of participation in the UDL, their reading comprehension rose by a year and two-thirds (1.7) on the nationally scale. Alumni Survey Results (RQ4). The 23 alumni respondents included 11 Blacks (African- Americans/ Caribbean Black), 6 Whites (Caucasian), 3 Hispanics, 2 Asians and 1 non-disclosed ethnicity. The average age of the respondents was 22 years old. All the study’s respondents graduated from high school; 74 percent continued their education into college and/or graduate school. Twelve identified their highest grade or program level completed as college, 3 as MA, 1 as PhD, and 1 as a professional school degree. When identifying their current occupation, the respondents indicated that 15 were currently college students, 2 were lecturers, 4 had careers in business, 1 was in the medical field, and 1 was currently unemployed. With the exception of one respondent who self-identified as “flakey with debate” rather than fully involved, all alumni who completed the survey indicated debate had enhanced their academic preparation for college. Two of the respondents indicated debate helped them strengthen their college applications, one because the analytical skills she learned from debate made her essays more “clever and well-written” and the other because she learned to tailor her arguments to explain why her personal profile fit with a particular college. One of the same two respondents went on to describe how debate also improved her college interviews. In her own words, “Debate also helped me in the interviews because I was able to respond fast to the questions they asked.” Beyond the application process, respondents listed fundamental skills that debate helped them to obtain that were useful once they had entered college: critical thinking, reading comprehension, public speaking, essay writing, study habits, annotation, problem solving, research methodologies, organization, time management, and practical sense. In the words of one respondent: “I am a much better reader, writer, and speaker. This helps across all college subjects to a greater or lesser degree.” The majority of respondents also reported that participation in UDLs helped bolster their academic confidence. One noted the impact before she even matriculated into college, when she stated, “It made me want to do well in school. It made me want to go to school and participate, ultimately helping me graduate valedictorian of my school and get into a good college.” Others commented on how their new sense of confidence gained from debate played out during their college years. One respondent reported, “I participate much more in class than I think I would otherwise,” with another proclaiming she is now a “champion hand raiser.” For several of the alumni respondents, debate helped reduce their fears of competing with their classmates. 10