THE EFFECT OF AN INTENSIVE TEACHER TRAINING ON THE ACCURACY OF SOCIAL, EMOTIONAL, AND BEHAVIORAL SCREENING RESULTS __________________________________________________________________" A Dissertation Presented to the Faculty of the Graduate School University of Missouri-Columbia __________________________________________________________________" In Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy __________________________________________________________________" by Kristy Warmbold-Brann University of Missouri-Columbia Drs. Matthew K. Burns and Stephen Kilgus, Dissertation Supervisors July 2017 © Copyright by Kristy Warmbold-Brann 2017 All Rights Reserved The undersigned, appointed by the dean of the Graduate School, have examined the dissertation entitled THE EFFECT OF AN INTENSIVE TEACHER TRAINING ON THE ACCURACY OF SOCIAL, EMOTIONAL, AND BEHAVIORAL SCREENING RESULTS Presented by Kristy Warmbold-Brann a candidate for the degree of Doctor of Philosophy, and herby certify that, their opinion, it is worthy of acceptance. Matthew K. Burns Ph.D. Department of Educational, School, and Counseling Psychology Stephen P. Kilgus, Ph.D. Department of Educational, School, and Counseling Psychology Cheryl A. Offutt, Ph.D. Department of Educational, School, and Counseling Psychology Chad A. Rose, Ph.D. Department of Special Education ACKNOWLEDGEMENTS The current project and my doctoral experience would not have been possible without so many people. I would like to recognize and thank the University of Missouri School Psychology faculty and graduate students for offering a remarkable training program and motivating me to strive for my best. I would like to thank my dissertation co-chairs, Drs. Matthew Burns and Stephen Kilgus. I cannot express my gratitude enough for your ongoing support and dedication to make the study possible. The project was left without an adviser and you both graciously offered your guidance without a second thought. As my adviser and research team leader, Dr. Burns also contributed greatly to my professional and personal growth and I greatly valued my time learning from him. Also, I would like to thank Dr. Melissa Maras for serving as a mentor over my five years at University of Missouri and fostering my interest in universal screening. Thank you for encouraging this project through countless screening conversations and applied work with schools. To my committee, I want to thank you for your willingness to discuss the project and think through logistical concerns and for always supporting the project. I very much enjoyed learning from you and the time spent discussing the project with you. Also, thank you to the Center for Social and Emotional Success for treating this study as your own and supporting the project. I know it was a lot of work and I sincerely appreciate everyone’s time, especially Dr. Stephen Kilgus, Dr. Katie Eklund, Crystal Taylor, and Amanda Allen for countless hours coordinating and handling logistical concerns. A special thank you to Jared Izumi, Lisa Aguilar, Kayla Kilpatrick, Rosie O’Donnell, Regan Riley, Deija McLean, Jennifer Connelly, and Mike Van Wie for making the project possible through diligent observations. Last but not least, I want to express my thanks to my incredible family and friends that filled my life with encouragement, laughter, and joy over these past five years. Thank you to my ii parents and sister for always supporting my studies and never questioning my love of school and learning. I also want to thank my informal family of the Southbrook house for helping me stay sane and keep challenges in perspective. To a doctoral candidate’s best friends, Ace and Brewer, thank you for always welcoming me home with a big smile and a wagging tail. My husband, Mark Brann, has been my greatest support during my graduate studies. Thank you for always making me laugh and encouraging my doctorate, even when it meant living apart and seeing me attached to my laptop. I cannot imagine these five years without you by my side serving as my cheerleader and informal advisor. iii TABLE OF CONTENTS CHAPTER I: INTRODUCTION ................................................................................................ 1 Statement of the Problem ............................................................................................................ 1 Purpose of the Current Study ...................................................................................................... 4 Definition of Key Terms ............................................................................................................. 5 Delimitations ............................................................................................................................... 6 CHAPTER II: LITERATURE REVIEW .................................................................................. 7 Behavioral Assessment ............................................................................................................... 7 Rating Scales ........................................................................................................................... 8 Systematic Direct Observation ............................................................................................... 9 Direct Behavior Ratings ........................................................................................................ 11 Synthesis ............................................................................................................................... 12 Rating Accuracy ....................................................................................................................... 13 Informant Discrepancy .......................................................................................................... 14 Teacher Differences in Ratings ............................................................................................. 17 Theoretical Frameworks ....................................................................................................... 19 Synthesis ............................................................................................................................... 20 Reducing Measurement Error ................................................................................................... 21 Types of Training .................................................................................................................. 23 Purpose ..................................................................................................................................... 26 Research Questions ................................................................................................................... 26 CHAPTER III: METHODS ...................................................................................................... 28 Participants ............................................................................................................................... 28 Measures ................................................................................................................................... 29 Social, Academic, and Emotional Behavior Risk Screening Scale (SAEBRS) ................... 29 Systematic Direct Observation (SDO) .................................................................................. 30 Training Conditions .................................................................................................................. 31 Frame of Reference Training with Familiarization Training ................................................ 31 Procedures ................................................................................................................................. 33 Data Analyses ........................................................................................................................... 37 CHAPTER IV: RESULTS ......................................................................................................... 42 Descriptive Statistics ................................................................................................................ 42 Comparisons to Observations of Behavior ............................................................................... 44 Academic Behavior ............................................................................................................... 44 Disruptive Behavior .............................................................................................................. 47 Impacts on SAEBRS Total Behavior Ratings .......................................................................... 48 CHAPTER V – DISCUSSION ................................................................................................... 50 Summary of Findings ............................................................................................................... 50 Impacts on SAEBRS Total Behavior Ratings .......................................................................... 53 Implications for Practice ........................................................................................................... 54 Implications for Theory ............................................................................................................ 55 Limitations and Future Directions ............................................................................................ 56 Conclusion ................................................................................................................................ 59 REFERENCES ............................................................................................................................ 60 iv APPENDICES ............................................................................................................................. 73 APPENDIX A ........................................................................................................................... 73 APPENDIX B ........................................................................................................................... 76 APPENDIX C ........................................................................................................................... 78 APPENDIX D ........................................................................................................................... 81 VITA ............................................................................................................................................ 83 v LIST OF FIGURES Figure 1. Study procedures 35 LIST OF TABLES Table 1. Grade Levels and Total Behavior Scores for Observation Participants 43 Table 2. Grade Levels and Total Behavior Scores for Screening Participants 44 Table 3. Observation Descriptive Statistics 45 Table 4. Total Behavior Descriptive Statistics by Condition 49 vi THE EFFECT OF AN INTENSIVE TEACHER TRAINING ON THE ACCURACY OF SOCIAL, EMOTIONAL, AND BEHAVIORAL SCREENING RESULTS Kristy Warmbold-Brann Drs. Matthew Burns and Stephen Kilgus, Dissertation Supervisors ABSTRACT The current study examined that effect of an intensive Frame of Reference teacher training on the accuracy of teacher-rated social, emotional, and behavioral screening results as compared to objective systematic direct observations of one to two students per classroom (n = 74). Teachers (n = 64) were randomized into an intensive Frame of Reference training or familiarization control condition. Results from multilevel analyses suggest no statistically significant improvement from an intensive teacher training on the difference scores of Academic Behavior and Disruptive Behavior. Nonparametric analyses were completed for Prosocial Behavior due to the lack of between-teacher difference and similarly found no improvement. The impact of the training on the Total Behavior results of all students (n = 1158) was also examined including the distribution, number of students identified per classroom, and Total Behavior Scores. It was hypothesized that the training would not impact Total Behavior Scores. The hypothesis was correct as the intensive training did not significantly alter the number of students identified, homogeneity of variance, or Total Behavior Scores. Limitations, directions for future research, and practical implications are reviewed in detail. vii CHAPTER I: INTRODUCTION Background Schools are tasked with the challenge of not only teaching academics but also supporting the development of behavior and social-emotional health (Satcher, 2000). There is a critical need to support student social, emotional, and behavioral health because approximately 14-20% of students experience behavior and/or mental health difficulties (Kessler Rc, 2005) and about half of Americans will meet the criteria for a mental illness diagnosis at some point in their lifetime (O'Connell, Boat, & Warner, 2009). In addition, the onset of social, emotional, and behavior disorders is usually in childhood and adolescence, and these students are at an increased risk for dropping out of school and having educational difficulties (Breslau, Lane, Sampson, & Kessler, 2008). Researchers have pinpointed a variety of risk factors that are connected to social, emotional, and behavioral disorders and it is believed that individuals have a higher likelihood of having a disorder when they are exposed to multiple risk factors (O'Connell et al., 2009). In addition, if children are not meeting developmental competencies, then they can be at-risk for social, emotional, and behavioral problems (Guerra & Bradshaw, 2008). Risk factors and developmental competencies can occur within the child or a variety of environmental contexts such as home, school, and community (Bronfenbrenner, 1977). It is crucial to find ways to identify and support these students in a proactive manner before major problems develop. Statement of the Problem School personnel can help prevent social-emotional difficulties with early intervention and universal screening for problems in order to identify students in need of supports (New Freedom Commission on Mental Health, 2003). Schools need a brief tool that provides an 1