School Mental Health https://doi.org/10.1007/s12310-020-09400-y ORIGINAL PAPER Teacher Ratings of Acceptability of a Daily Report Card Intervention Prior to and During Implementation: Relations to Implementation Integrity and Student Outcomes Erin Girio‑Herrera1  · Theresa E. Egan2 · Julie Sarno Owens3 · Steven W. Evans3 · Erika K. Coles4 · Alex S. Holdaway2 · Clifton S. Mixon5 · Hannah D. Kassab3 Accepted: 17 November 2020 © Springer Science+Business Media, LLC, part of Springer Nature 2021 Abstract The goals of the study were to (a) examine teacher-reported acceptability of a daily report card (DRC) intervention for a student in their classroom prior to and during implementation; (b) examine factors that predict acceptability; and (c) explore the relations between teacher-reported acceptability, student and teacher characteristics prior to implementation, implementa- tion integrity (treatment dose, adherence, and teacher competence), and student outcomes. Participants were 39 elementary school teachers and 39 students with or at risk of attention-deficit hyperactivity disorder (ADHD). Teachers were asked to implement the DRC for up to 16 weeks with consultation support provided by research team staff every other week. Teachers completed acceptability ratings about the DRC prior to and after two months of implementation. This multi-method assess- ment using correlation and regression analyses revealed that although acceptability ratings prior to implementation were related to teacher knowledge of ADHD, they were was not related to acceptability ratings during implementation, integrity, or student outcomes. Student’s initial positive response to the intervention (i.e., the magnitude of improvement in DRC target behaviors) was associated with higher acceptability ratings during the intervention. Greater increases in acceptability over time were associated with greater DRC dose (i.e., teacher compliance to procedures and longer DRC duration). Greater duration of implementation and responding appropriately to rule violations were associated with greater student achievement of DRC goals. Implications for interpreting acceptability ratings and for understanding factors related to implementation and outcomes are discussed. Keywords Acceptability · Teacher · Implementation · Attention-deficit/hyperactivity disorder (ADHD) · Intervention · Daily report card Introduction Attention-deficit hyperactivity disorder (ADHD) affects * Erin Girio-Herrera 5–11% of school-aged children (Merikangas et al., 2010; [email protected] Wolraich et al., 2014). Thus, most elementary classrooms contain one to two students with ADHD (Fabiano et al., 1 Department of Psychology, Towson University, Towson, 2013a, b). Relative to non-ADHD peers, students with MD 21252, USA ADHD often experience lower academic achievement 2 Department of Child and Adolescent Psychiatry (Volpe et al., 2006), higher rates of absenteeism (Classi, Mil- and Behavioral Sciences, Children’s Hospital of Philadelphia, Philadelphia, PA, USA ton, Ward, Sarsour, & Johnston, 2012) and grade retention 3 Department of Psychology, Ohio University, Athens, OH, (Loe & Feldman, 2007), and more conflict with peers and USA teachers (Greene, Beszterczey, Katzenstein, Park, & Goring, 4 Department of Psychology, Florida International University, 2002; Hoza, 2007). Students with ADHD are more likely to Miami, FL, USA use special education services and receive more disciplinary 5 Department of Pediatrics, Ochsner Hospital for Children, referrals than their non-ADHD peers (Robb et al., 2011). Ochsner Health System, New Orleans, LA, USA Given the negative effects of ADHD on the student and the Vol.:(0112 33456789) School Mental Health school (e.g., financial costs, teacher stress), it is important of the school environment: the external environment, organi- for educators to be equipped with effective interventions to zation, intervention, and interventionist. Listed among the address the needs of these youth and for these interventions factors related to the interventionist are many that involve to be implemented with integrity. the interventionists’ perceptions. For teachers involved in Classroom-based behavior management interventions are the implementation of the DRC, this could include their per- classified as well-established interventions for elementary ceived need for intervention, effectiveness of intervention, school students with ADHD (Evans, Owens, Wymbs, & role in implementation, and self-efficacy. Ray, 2018). The most widely studied of these is the daily One factor that may affect teachers’ implementation report card (DRC) intervention. When using a DRC, teach- integrity is their perception of intervention acceptability, that ers identify and define two to three target behaviors that is, the extent to which the intervention is viewed as appro- cause impairment for the student, track those behaviors for priate, fair, and reasonable for the problem (Kazdin, 1981). one week, and use these baseline data to establish the initial, Witt and Elliott (1985) proposed a model containing four achievable goals for each target behavior (e.g., completes elements (i.e., intervention acceptability, use, integrity, and 75% of daily math problems, raises hand before speaking effectiveness) that are sequential and reciprocal. They sug- with four or fewer violations). During the day, the teacher gest that high intervention acceptability is associated with tracks each behavior and provides feedback to the student high intervention use, which is subsequently associated with about progress toward each goal. At the end of the day, DRC high intervention integrity (i.e., adhering to recommended performance is reviewed with the student by teachers and procedures). High intervention integrity is proposed to be parents and contingent privileges are provided. Using shap- related to high intervention effectiveness, which in turn ing procedures, each behavior is modified until it moves into facilitates high intervention acceptability, thus creating a the typical range for the student’s age. dynamic interactive cycle. In this way, acceptability can be The DRC is effective for students with ADHD in general conceptualized as both a predictor of use and integrity, and education (e.g., Owens et al., 2012) and special education an outcome of intervention effectiveness. Reimers, Wacker, (e.g., Fabiano et al., 2010) classrooms, is feasible for use and Koeppl (1987) expanded this model to include treatment over several months (e.g., Owens et al., 2012; Owens, Mur- knowledge and environmental disruption (i.e., interruption phy, Richerson, Girio, & Himawan, 2008), and is effective to typical activities) as additional factors influencing treat- in modifying academic and behavioral problems (e.g., Pyle ment acceptability. These models are foundational in the & Fabiano, 2017; Vannest, Davis, Davis, Mason, & Burke, school-based literature and are echoed in the implementation 2010). Further, there are incremental benefits of the DRC science literature (see Proctor et al., 2009 for review). with each month of the intervention over four months (Hold- Often treatment acceptability has not been evaluated in away et al., 2020; Owens et al., 2012). Despite this evidence, the context of environmental disruption. However, when there is variability in the integrity with which teachers teachers are asked to report on their acceptability of an inter- implement this (and other) classroom interventions which vention, it is valuable to capture their perceptions against has implications for intervention effectiveness. For exam- the backgroup of their countless other responsibilities. One ple, two studies found that, on average, teachers adhered study showed that among teachers implementing student to recommended DRC procedures on 77% of school days; interventions (classroom and individual), those who dem- yet the range in both studies was wide (0–98% in Fabiano onstrated acceptable integrity (between 80 and 100% on an et al.; 10–100% in Owens et al.). Given this variability, and integrity checklist) were more likely to evaluate the inter- the well-established link between intervention integrity and ventions to be suitable and convenient; whereas those who student outcomes (e.g., Conroy et al., 2015; Noell, Gresham, implemented the interventions with very high (100%) or low & Gansle, 2002), there is a need to better understand factors integrity (less than 80%) rated the interventions less suit- that predict integrity. able and convenient (Harrison, State, Evans, & Schamberg, Treatment integrity is a critically important methodologi- 2016). These findings suggest that the extent to which teach- cal consideration in ensuring validity in treatment outcome ers pushed themselves (intensively, moderately, or slightly) research and implementation science. It is a multifaceted to implement the intervention in light of competing priori- concept with several definitions and models and is often ties, may have an impact on the teachers’ perceptions of overlooked in treatment outcome research (Cox, Martinez, treatment acceptability. Examining acceptability within the & Southam-Gerow, 2019). A description and exploration framework of these conditions offers a more realistic picture of all factors associated with treatment integrity is beyond of teachers’ true perceptions of interventions. the scope of this manuscript; however, the complexity of Previous studies have examined teachers’ acceptability of integrity is noteworthy. Hagermoser Sanetti and Kratochwill a DRC; however, the conclusions drawn are limited due to (2009; Table 2) highlight several factors proposed to influ- the use of vignettes or hypothetical scenarios and the lack of ence treatment integrity across four levels within the context measurement of acceptability during implementation (e.g., 1 3 School Mental Health Gresham & Lopez, 1996). Further, despite calls to action in ratings of acceptability remain high. However, none of these the general acceptability literature (e.g., Nastasi & Truscott, studies examined the association between the acceptabil- 2000), no studies have examined intervention acceptability ity and integrity or student outcomes. Thus, it is not clear of the DRC in relation to factors existing prior to implemen- whether high acceptability is a predictor of teachers’ integ- tation, implementation integrity, and outcomes. Measuring rity or an outcome of teachers’ experiences with the DRC, acceptability and teacher and student factors at multiple or both, as hypothesized by Witt and Elliott (1985). Indeed, timepoints expands on previous studies by conceptualizing Gresham and Lopez (1996) once argued that “while inform- acceptability as a dynamic factor that is potentially related ative… pretreatment acceptability may not correspond to to prior factors and may be associated with subsequent what consumers might tell us about the acceptability of implementation and student outcomes. The current study treatments after they have tried them” (p. 213). It may be addresses these limitations by examining teachers’ ratings more important to determine acceptability after the use of of acceptability of a DRC prior to and during DRC imple- an intervention, as this experience and observation of child mentation, and the relationships between DRC acceptability, response likely influences continued use and perceptions of teacher characteristics and student characteristics, integrity, the intervention. and outcomes. Factors Affecting Acceptability and Implementation Intervention Acceptability Research Given that there is some variability in teachers’ accept- Studies measuring teachers’ reports of acceptability of non- ability ratings for a given intervention, prior research has DRC treatments suggest that teachers demonstrate higher investigated characteristics of the teacher and student prior integrity and observe greater change in disruptive student to implementation as potential predictors of intervention behavior when they implement interventions they find more acceptability. First, consistent with Reimers and colleagues’ acceptable (Andersen & Daly, 2013). Johnson et al. (2014) (1987) model, two reviews highlight studies that found a demonstrated that teachers who implemented a preferred positive relation between teacher knowledge (e.g., of stu- or acceptable intervention adopted the intervention more dent problems or behavioral principles) and intervention quickly, sustained higher implementation quality independ- acceptability (e.g., see Elliott, 1988; Han & Weiss, 2005 ent of coaching, and were more likely to continue imple- for review). Thus, we included a measure of knowledge of menting the intervention following study completion, as ADHD and hypothesized that it would be positively related compared to teachers who implemented an assigned inter- to higher acceptability ratings and higher integrity and use vention. Thus, improving teachers’ acceptability of inter- (Vereb & DiPerna, 2004). ventions and/or designing interventions in collaboration Second, some studies have found that teachers’ years of with teachers (so that they are acceptable) may be effective experience and highest degree earned are positively related strategies to enhance intervention adoption and to improve to acceptability of some classroom interventions (Girio & integrity and related student outcomes. Owens, 2009; Vereb & DiPerna, 2004), whereas others Previous studies examining teachers’ treatment accept- have found no relation (Pisecco et al., 2001; Power et al., ability for behavioral interventions document that teachers 1995). However, all of these studies occurred in the context generally find most evidence-based and promising inter- of hypothetical vignette methods. Further, with other class- ventions are acceptable (Briesch, Briesch, & Chafouleas, room interventions, years of teaching and highest degree 2015; Elliott, Witt, Galvin, & Peterson, 1984; Girio & earned were unrelated to the number of times the strategy Owens, 2009; Power, Hess, & Bennett, 1995). Specific to was used (Domitrovich et al., 2015). Given this variability the DRC, Girio and Owens (2009) reported that elementary across studies, years of experience and degree earned were school teachers endorsed the DRC as acceptable, and more included but directional hypotheses were not made. acceptable than other interventions for addressing disruptive Lastly, in addition to teacher-level factors, the most sali- behavior (e.g., time out, medication, social skills), which ent student-level factor associated with teacher ratings of is consistent with prior work (Pisecco, Huzinec, & Curtis, acceptability is severity of student problems. Multiple stud- 2001; Power et al., 1995). However, most studies have exam- ies have found that teachers view behavioral interventions to ined teachers’ acceptability under hypothetical conditions be more acceptable and reasonable when student problems (e.g., rating acceptability after reading vignettes) rather than are severe (as compared to when student problems are mild during implementation of an actual intervention (Gresham or moderate), presumably because milder problems may not & Lopez, 1996). Among the few studies that have exam- warrant the effort required to implement the intervention ined teacher report of acceptability of the DRC before or (Elliott et al., 1984; Martens, Witt, Elliott, & Darveaux, after actual implementation (e.g., Murray, Rabiner, Schulte, 1985). However, most of these studies used vignettes and & Newitt, 2008; Williams, Noell, Jones, & Gansle, 2012), there are a few studies with contrary findings (see Elliott, 1 3 School Mental Health 1988 for review). Thus, examination of student severity study). Analyses of available data revealed that teachers under implementation conditions is warranted. We assessed included in the current sample did not differ from those student impairment in multiple domains of functioning and excluded with regard to teacher or student gender, site, years hypothesized that greater severity would be associated with of teaching experience, student impairment, DRC accept- higher acceptability. ability ratings, DRC adherence, competence ratings, or stu- dent initial response to the DRC. Excluded teachers were Current Study more likely (than included teachers) to be in third or fifth grade and, on average, implemented the DRC for fewer days. The first aim examines teacher-reported acceptability of a Included teachers were women (94.9%) and identified as DRC intervention prior to and during implementation of a Non-Hispanic White (53.8%) or Hispanic (any race; 43.6%). DRC for a student in their classroom. It was hypothesized Included teachers had an average of 14.88 years (SD = 8.72) that teacher-reported acceptability of the DRC would be of teaching experience. Most (64%) had a master’s degree high (Girio & Owens, 2009). The second aim examines fac- or higher. tors that predict acceptability. The third aim explores the Target students were 39 elementary school students relation between teacher report of acceptability, baseline (76.9% male; 51.3% Hispanic). Most target students (92.3%) student and teacher characteristics, implementation integ- met criteria for ADHD (71.8% combined presentation; rity (dose, adherence, and teacher competence), and student 17.9% inattentive presentation; 2.6% hyperactive/impulsive outcomes. It was expected that acceptability both prior to presentation) and the remaining 7.7% were at risk of ADHD and during treatment would be related to implementation (at least four symptoms plus teacher-rated impairment). integrity and student outcomes. Further, consistent with the The DRC is effective for a variety of presenting problems; interactive, dynamic models (Witt & Elliott, 1985; Reimers thus, we allowed subclinical symptoms as long as there was et al., 1987), it was hypothesized that initial positive student impairment. Students had an average IQ estimate of 98.24 outcomes would be positively related to later acceptability (SD = 13.43), as assessed by the Wechsler Abbreviated and implementation integrity. Scales of Intelligence, Second Edition (Wechsler, 2011). Family socioeconomic status was low to middle class (15.4% had a household income of under $15,000, 58.9% had an Method income between $15,000 and 49,999; and 18% were above $50,000; 7.7% did not report income). Per parent report at Participants intake, 2.6% had been diagnosed with a learning disability and 23.1% had a medication prescription for ADHD. Eight schools participated across two sites. In Ohio, the five Procedures participating schools had an average of 377 students and 16 general education teachers per school, with 12–29% of students receiving special education services and 35–75% The study was conducted at two universities and proce- receiving free or reduced lunch services. In Florida, the three dures were approved by the Institutional Review Boards at participating schools had an average of 1024 students and both and within all participating school districts. A com- 50 general education teachers with 4–11% receiving special plete description of procedures can be found in Owens et al. education services and 76–95% receiving free or reduced (2017). See Fig. 1 for a list of study constructs and the time- lunch services. The racial makeup of schools was primar- line for data collection. All general education teachers in ily Caucasian (range 90–98%) in Ohio and predominantly each elementary school were invited to a 3-h workshop con- Latinx in Florida (range 94–98%). ducted by the investigators that focused on best practices in Teacher participants were 39 elementary school teachers general classroom management strategies and the DRC. At (19 from Ohio, 20 from Florida) teaching grades K through the end of the workshop, teachers completed the question- 5 who were participating in a multi-site consultation study naires described below. Teachers interested in participating designed to facilitate teachers’ implementation of effective in consultation were required to identify one student with classroom management strategies and a DRC intervention or at risk of ADHD; consent was required by teacher and (masked for review). Teachers represented in the current parent, and assent was required by the student. Inclusion study are those who completed all measures prior to and criteria for being a target student were the following: (a) during DRC implementation. Nineteen teachers (not part enrolled in a general education classroom (K-5) for at least of the 39) were excluded because they had not completed 50% of the day, (b) IQ estimate that fell in or above the 90th the acceptability measure at one of the two time points (in percentile confidence interval for a score of 80, and (c) met five cases because the student moved before the second time diagnostic criteria for DSM-IV ADHD or were at risk of point; in three cases because the teacher withdrew from the ADHD. ADHD was defined as the presence of six or more 1 3 School Mental Health Fig. 1 Temporal representation of study constructs symptoms of inattention and/or hyperactivity/impulsivity as Witt, Singletary, & VanDerHeyden, 2007; Noell, Witt, Gil- reported by parents on the Children’s Interview for Psychi- bertson, Ranier, & Freeland, 1997). In the multi-component atric Syndromes-Parent Version (P-ChIPS; Fristad, Teare, condition, consultants followed the problem-solving process Weller, Weller, & Salmon, 1998) or the parent or teacher described above, but also assessed and attempted to address version of the Disruptive Behavior Disorders Rating Scale possible barriers to integrity using the knowledge, skills, and (Pelham, Gnagy, Greenslade, & Milich, 1992), and teacher- beliefs components (Owens et al., 2017). Both conditions rated impairment as defined by a rating of at least 3 on the focused on creation of the DRC, use of the DRC, general Impairment Rating Scale (Fabiano et al., 2006). Information classroom management strategies, receipt of performance obtained from the P-ChIPs helped to rule out other disorders feedback, discussion of implementation, and problem- as sources of ADHD symptoms and to assess symptom chro- solving. Sessions were 30 min to 1 h and conducted before, nicity. At-risk status was defined as four or more symptoms during, or after school. Although adequate differentiation and teacher-rated impairment. Students were excluded if a of conditions and an equal number of consultation sessions previous diagnosis of autism spectrum disorder, bipolar dis- across conditions was achieved (Owens et al., 2017), for the order, or intellectual disability was reported by the parent. current study, teachers were combined across conditions. Once a target student was identified, teachers were asked Teachers in the two conditions did not differ on acceptabil- to implement the DRC for 16 weeks. Teachers completed ity prior to the intervention implementation (condition 1: the questionnaires described below a second time after M = 5.18, SD = .62; condition 2: M = 5.26, SD = 1.02; two months of DRC intervention implementation. Teachers t(37) = − .275, p = .79). were paid for attending the inservice and completing ques- Measures tionnaires, but did not receive compensation for participat- ing in consultation sessions or for implementation of any classroom management practices. Student Impairment Consultation Procedures The teacher version of the Impairment Rating Scale (IRS; Fabiano et al., 2006) assesses teacher perceptions of student For the purposes of a clinical trial, stratified random sorting functioning on a 7-point scale that ranges from 0 (No prob- was used to assign teachers to two consultation conditions lem, Definitely does not need treatment) to 6 (Extreme prob- (Owens et al., 2017). The standard condition was designed lem, Definitely needs treatment or special services). This to represent best practices in school psychology. It followed was completed at the point of student referral to the project. a problem-solving process (Frank & Kratochwill, 2014) and The domains assessed include relationship with peers, rela- included brief performance feedback that mirrored best prac- tionship with teacher, academic progress, the classroom in tice procedures reported in previous research (Gilbertson, general, self-esteem, and overall. With elementary school 1 3 School Mental Health samples, the measure has respectable cross-informant reli- (Girio & Owens, 2009; Martens et al., 1985). In the cur- ability, convergent and divergent validity with other impair- rent sample, internal consistency estimates were .96 prior ment scales, and predictive validity in identifying students to and .91 during implementation. with ADHD diagnoses (Fabiano et al., 2006) when scores are three or higher. Initial DRC Response Teacher Demographic Information To examine the student’s initial response to the DRC, we calculated individual effect sizes (standard mean differ- Teachers were asked to provide the number of years they ence; SMD) that represented the cumulative benefit of the have taught and their highest degree earned. DRC at the end of Month 1 and the end of Month 2. This 2-month time frame was selected based on a previous study Teacher Knowledge of ADHD that demonstrated that (a) large effects could be detected after one month of implementation and (b) responders to The teacher knowledge of ADHD questionnaire is a 24-item the intervention were highly distinguishable from non- true/false/don’t know measure that assesses teacher knowl- responders (Owens et al., 2012) after two months of imple- edge of ADHD and best practices in the treatment of ADHD. mentation. For the cumulative effect size (ES), the SMD Responses were coded as correct or incorrect. Don’t know represents the difference between the mean of a follow-up was coded as incorrect. A total percent correct was calcu- period (i.e., Month 1, Month 2) and the mean of the period lated. The measure was developed by the authors of Owens prior to implementation (i.e., baseline tracking) divided by et al. (2017) clinical trial and was inspired by the meas- the standard deviation of the period prior to implementa- ure developed by Jones and Chronis-Tuscano (2008). The tion. By this definition, SMD can be interpreted as the mean measure has demonstrated sensitivity to change as a func- improvement of a participant at a follow-up period, com- tion of participating in a workshop focused on ADHD and pared to the baseline period, as a function of the variability classroom management (Owens, Coles, & Evans, 2014). during the baseline period. This procedure is consistent with Because this measure was completed after the workshop, other studies that examined the incremental benefit of the scores reflect that all teachers were given an equal opportu- DRC (Holdaway et al., 2020; Owens et al., 2012) and differ- nity to have foundational knowledge about ADHD. ent medication doses for students with ADHD (e.g., Evans et al., 2001). For this study, the average ES from Month 1 Intervention Acceptability and Month 2 for the student’s first two DRC targets was calculated, as all students had at least two DRC targets and Intervention Rating Profile-10 Item Version (IRP-10; these typically represented the teachers top concerns. Power et al., 1995) was used to assess teacher percep- tions of DRC acceptability prior to implementation and two months after implementing the DRC. Items are rated DRC Dose on a 6-point scale that ranges from 1 (Strongly Disa- gree) to 6 (Strongly Agree). Ratings for each item are DRC implementation requires the teacher to give student mean-averaged to yield a total score reflecting a single feedback when a rule violation occurs (e.g., Carlos, that’s an dimension of acceptability. Higher scores indicate higher interruption) and make a tally for that rule violation on the acceptability of that treatment. The measure includes a DRC. Teachers were asked to either insert these data into a few general questions such as, “I liked the procedure used website that produced graphs of performance or give these in this intervention” and “I would suggest the use of this data to the consultant who assisted with data entry. Dose was intervention to other teachers.” However, the majority of calculated in two ways (compliance and duration) based on items are specific to the student such as, “This would be this data. Compliance was defined as the number of days in an acceptable intervention for [Child’s] school difficul- which DRC data were submitted divided by the number of ties;” “Overall, this intervention would be beneficial for days data could have been submitted (i.e., all days for which [Child];” “This intervention would not result in negative the student was present at school during the 2-month DRC side effects for [Child]:” and “[Child’s] school problems implementation period). This was calculated for the first two are severe enough to warrant use of this intervention.” targets on the student’s DRC, then averaged. For duration, The IRP-10 has excellent internal consistency with alpha the number of days between DRC initiation and the last day coefficients ranging from .95 to .97 (Power et al., 1995) of DRC data submission was calculated. Data on compliance and has evidence of discriminant validity, as it can iden- and duration were gathered across the academic year and tify interventions of varying acceptability among teachers serve as dependent variables. 1 3 School Mental Health DRC Adherence were conducted to obtain a baseline assessment of each teacher’s competence in classroom management. Compe- Teachers were observed using the Student Behavior-Teacher tence was evaluated in several domains (e.g.,. response to all Response Observation Rating System (SBTR; Pelham, inappropriate behaviors, response to DRC-related behavior, Greiner, & Gnagy, 2008). This is a systematic class-wide and global competence). Observers considered facets within observation system developed using a behavior theory each domain (e.g., timing, specificity, tone of voice, consist- framework and intended to capture discrete student–teacher ency) and assigned a competence rating on a 10-point scale. interactions in preschool and elementary classrooms. Previ- For the current study, competence in relation to teachers’ ous studies have shown SBTR to have adequate interrater response to inappropriate behavior and DRC-related behav- reliability and convergent validity (Fabiano et al., 2013; iors (averaged across Month 1 and Month 2 observations) Vujnovic, Holdaway, Owens, & Fabiano, 2014), as well as were used as predictor variables. The ICC(1,k) were ≥ .88, sensitivity to change as a function of intervention (Fabiano respectively. et al., 2010; Owens et al., 2017). The SBTR observation manual includes definitions, and inclusion and exclusion examples for coding student violations of seven common DRC Outcomes classroom rules (i.e., be respectful, obey adults, work qui- etly, use materials appropriately, remain in seat, raise hand We examined DRC outcomes in two ways. First, we exam- to speak, stay on task), and the teacher’s response to each ined the incremental SMD effect size for Month 3 (i.e., the violation (i.e., coded as appropriate, inappropriate, or no month following completion of acceptability ratings). The response). All definitions are available upon request from incremental SMD for Month 3 represents the difference the first author. SBTR observers obtained frequency counts between the mean of the data during the last two weeks of of (a) all rule violations by the target student, (b) how the Month 3 and the mean of the data during the last two weeks teacher responded to each of those types of violations, (c) of Month 2 divided by the standard deviation of the period all DRC violations, and (d) how teachers responded to each prior to implementation. The SMD can be interpreted as the of those types of violations (i.e., responded by labeling the average improvement during Month 3 compared to Month 2, DRC behavior or not, Carlos, that’s an interruption). as a function of the variability at baseline. This is an indica- Two variables from the SBTR observations were used to tor of student improvement after acceptability ratings were represent adherence for the current study: (a) teacher per- given. Second, we examined a global indicator of the stu- cent appropriate response to student rule violations and (b) dent’s success with the DRC, namely the student’s overall teacher percent appropriate response to DRC violations. To achievement of all DRC goals over the duration of imple- calculate these variables, the total number of appropriate mentation. We included all goals, so that this represented the teacher responses to each violation for a given observation overall achievement with this intervention. It was defined as was divided by the total number of the respective violations the number of days the student met their DRC goal divided by the student for that observation period. These percentages by the number of days DRC data was implemented. For were captured for Month 1 and Month 2 and then averaged. example, if a student had a goal of “five or fewer interrup- Both variables serve as a predictor variable for the analyses. tions” and the student achieved five or less interruptions Observers were trained to reliability on the SBTR. They on 30 of 50 days DRC data were submitted, the student’s attended an initial training, were required to pass (100% achievement of this DRC target would be 60%. Then, this accurate) a written definitions test, (100% accuracy) coding percentage was averaged across all DRC target behaviors. of 2–5-min video clips, and achieve at least 80% reliabil- ity across all coded behaviors in a classroom with a master observer. Maintenance of reliability was checked across the Results year. Interobserver assessments were conducted for 24% of all observations in the clinical trial. To assess the inter- Aim 1: Acceptability Prior to and During rater reliability, calculations were completed for Intraclass Implementation Correlations (ICC) of type 1 for average of k raters (that is ICC(1,k). Across all frequency count variables, the ICC(1,k) ranged from .88 to .94 with an average of .93. On average, teacher-reported acceptability prior to imple- mentation was 5.23 (range 1–6) and during implementa- DRC Competence tion was 5.20 (range 3–6). Descriptive statistics for all other study variables can be found in Table 1. As shown in Once a target child was identified and teacher consent was Table 1, all variables had adequate variability for inclusion obtained, at least two classroom observations (using SBTR) in the regression models. 1 3 School Mental Health Table 1 Descriptive statistics for study variables response to treatment (average effect size for behavioral tar- gets 1 and 2 across Month 1 and 2). Pre-implementation Variable M (SD) Range (N = 39) acceptability was entered hierarchically into the first step, and the other predictors were entered together in the second Acceptability step (see Table 3). The first step (pre-implementation DRC IRP pre-intervention 5.23 (.86) 1–6 acceptability) was not significant. The second step was sig- IRP during intervention 5.20 (.74) 3–6 nificant, F(4) = 5.83, p < .01, accounting for 41% of the vari- Baseline variables ability in DRC acceptability during implementation. Nota- Years teaching 14.88 (8.72) 1–36 bly, student’s initial response to treatment (effect sizes) was % with Master’s degree or higher 64.0 a significant predictor (p < .01), and the beta (.44) indicated ADHD knowledge (% correct) 83.9 (10.9) 63–100 that larger effect sizes (greater improvement over Month 1 IRS-Tch–student relationship 3.62 (1.44) 0–6 and 2) were associated with higher acceptability ratings at IRS-Academic 4.54 (1.55) 0–6 Month 2 (see Table 3). ADHD knowledge and teacher-rated IRS-Classroom 4.08 (1.61) 0–6 student impairment in classroom functioning (prior to imple- Initial DRC response mentation) were marginally significant (p < .07). Avg Cum ES Targets 1 & 2 @ Mos 1 .67 (1.07) − 2.77–2.90 Of note, although the dose variables (compliance and & 2 duration) correlated with acceptability during intervention, DRC dose they were not included in the model as these variables were Compliance for targets 1 & 2 (%) 88.7 (10.5) 54–100 measured across the academic year. Further, one adherence Duration (number of days) 55.61 (19.34) 18–87 variable (teacher percentage of responding appropriately) DRC adherence was correlated with acceptability. However, adherence and Avg respond appropriately Mos 1 & 2 37.36 (26.82) 0–100 (%) competence in Months 1 and 2 could be considered either Avg label DRC Mos 1 & 2 (%) 55.61 (33.55) 0–100 a predictor of acceptability during implementation or a DRC competence response to acceptability. Including the responding appro- Avg respond Inappr Behav Mos 1 and 2 5.83 (2.70) 3.50–9.25 priately variable as a predictor did not modify the results Avg respond DRC Behav Mos 1 and 2 6.99 (1.25) 1–10 above; thus, the simplest model was presented. DRC outcome Aim 3: Relationship between Acceptability, Avg incremental ES targets 1 and 2 @ .45 (.97) − 1.5–2.45 Implementation, and Student Outcomes Mo 3 Overall achievement of DRC goals (%) 65.79 (18.38) 19–97 This aim was examined by exploring student outcomes (the N = 39 student’s initial and overall improvement and achievement IRP Intervention Rating Profile-10, IRS Impairment Rating Scale, Tch Teacher, DRC daily report card, ES standard mean difference effect of DRC goals) and the relation between teacher report of size, Avg average, Cum cumulative acceptability, baseline teacher characteristics (e.g., knowl- edge, years of experience, highest degree earned), baseline student characteristics (e.g., initial severity of impairment), Aim 2: Factors that Predict Acceptability During and implementation integrity (dose, adherence, and teacher Implementation competence). Two variables had significant correlations with incremen- First, we examined correlations between all theoretically tal improvement in Month 3: the number of years teaching relevant variables, i.e., teacher report of acceptability, stu- and DRC compliance (see Table 2). This model, F(2) = 5.50, dent and teacher characteristics prior to implementation, and p < .01, accounted for 31% of the variance in the incremental integrity variables (dose, adherence, and competence and improvement in Month 3. The number of years teaching was student outcomes). See Table 2. Variables that are significant a significant predictor (Beta = − .40, p < .05), indicating less (p < .05) were included in each regression model and are years of experience was associated with more improvement described below. in Month 3 (see Table 4). The first hierarchical linear regression model examined The following variables had significant correlations factors that predict DRC acceptability during implementa- with overall achievement of DRC goals: DRC acceptabil- tion. The following baseline and early implementation fac- ity during treatment, student initial response to treatment tors correlated significantly with DRC acceptability during (effect sizes), compliance, duration, and adherence (i.e., implementation and were included as possible predictors: responding appropriately to rule violations). Because over- teacher knowledge of ADHD, teacher-reported student all achievement of DRC goals is partially a function of ini- impairment in classroom functioning, and students’ initial tial response to treatment, this variable was not included 1 3 School Mental Health vg o AM 6. S 1E3 s ReC 15. DRBeh s Rep 14. InapBeh g v 13. ALab DRC g v 12. AResp Appr – C R D 11. Dur – .20 C R 10. DComp – .58** .16 vg Mos 2 * * 9. AES 1 & – .46* .37* .46* S Rss 8. ICla – .10 .16 .06 .17 S 7. IRAcad – .12 − .12 .03 − .20 − .23 d 6. IRS Tch-Stu – .48** .25 .01 − .15 − .22 − .02 D H Dw 3 3 Ao 0 * 0 5. Kn – .13 − . .34 − . .09 .11 .01 st e 4. HigheDegre – .10 − .15 .09 .12 − .24 .08 27− . − .00 s e y variabl 3. Yrs Teach – .47** − .02 .01 − .06 .11 .09 − .02 − .16 .09 d u etween st 2. IRP During – .04 .11 .33* − .10 − .10 .41** .46** .40* .51** .34* b s e Table 2 Correlation 1. IRP Pre Acceptability 1. IRP Pre– 2. IRP − .02DuringBaseline variables .013. Years Teach-ing 4. Highest .13Degree .38*5. ADHD Knowl-edge − .106. IRS-Tch–student − .097. IRS-Aca-demic .038. IRS-Class-roomInitial DRC respons .019. Avg ES Target 1 & 2 M1 & M2DRC dose 10. Com-− .07pliance 11. Dura-.12tionDRC adherence .0712. Avg % Resp Appr M1 & M2 1 3 School Mental Health Avg Mo ffer- 16. ES 3 – .29 an di e m 15. Res DRC Beh – .06 .18 standard 14. Res g Inapp Beh – .71** .27 .03 ESport card, 13. AvLab DRC – .41* .14 − .06 − .30 daily re C g R 12. AvResp Appr − .06 .26 .33* .06 .46** Doom, C ssr 11. DRDur − .30 .00 .05 .16 .58** Class cla RC mic, 10. DComp − .02 .22 .34* .41* .35* d acade 9. Avg ES Mos 1 & 2 .05 .26 .34* − .15 .60** Acaudent, st d S u 8. IRClass − .08 − .07 − .06 .35 .16 Sther, c a e 7. IRS Acad .04 .04 − .04 − .04 − .38* Tchge, t d 6. IRS Tch-Stud .28 .17 .04 .13 − .12 ow knowle D Kn 5. ADHKnow − .00 .39* .14 .08 − .03 g Scale, n 4. Highest Degree − .11 .23 .30 − .22 − .30 airment RatiM3 Month 3 3. Yrs Teach − .07 − .18 − .16 − .50** − .20 IRS10, Imp2 Month 2, Table 2 (continued) 1. IRP 2. IRP PreDuring − .07.0613. Avg % Label DRC M1 & M2DRC competence .30.2114. Avg Resp Inappr Behav M1 & M2 .13.2415. Avg Resp DRC Behav M1 & M2DRC outcome − .11.0616. Avg ES Tar-get 1 & 2 M3 17. DRC − .16.44**Achieve N = 39IRP Intervention Rating Profile-M1Mence effect size, Month 1, +p *p **p < .07; < .05; < .01 1 3