THE IMPACT OF ERROR ON OFFENDER RISK CLASSIFICATION By Aaron K.T. Ho A Dissertation submitted to the Graduate School-Newark Rutgers, The State University of New Jersey In partial fulfillment of the requirements For the degree of Doctor of Philosophy Graduate Program in Criminal Justice Written under the direction of Dr. Todd Clear And approved by ___________________ ___________________ ___________________ ___________________ Newark, New Jersey May, 2013 ©2013 Aaron K.T. Ho ALL RIGHTS RESERVED ABSTRACT The Impact of Error on Offender Risk Classification By Aaron K.T. Ho Dissertation Director: Todd Clear In criminal justice, offender risk classification seeks to divide individuals into different groups, normally so that varying levels of program treatment, custody, or supervision can be effectively and optimally allocated. The goal of effectively separating offenders based on prearranged criteria, however, is often thwarted by error problems, resulting in the misclassification of individuals. How the initial error problems eventually translate into final misclassification is not completely understood. Thus, the dissertation attempts to model the effects of error on the tolerance of offender risk classification instruments. Specifically, different properties and characteristics of classification devices are analyzed to understand their impact on the transfer of error from initial to final classification phases. Suitable risk data and instruments that would facilitate the testing of all proposed research questions and hypotheses in the current study are not readily available. This is because, in order to explore the different facets of the proposed inquiries, specific situations are requisite- and these particular situations may be easier tailored into a fabricated data than to be found in the real world. Thus, relying on both conceptual data and actual risk data, random and systematic error are simulated and injected into each ii risk instrument to gain insight onto how unreliability and invalidity statistically impact classification. The risk data are engineered using Monte Carlo Simulation: construction methods making use of random draws from an error distribution and multiple replications over a set of known parameters. This methodology is particularly relevant in situations where the only analytical findings involve asymptotic, large-sample results. Monte Carlo Simulations enables the construction of multiple datasets in a “laboratory setting” that would simulate data in the real world. This allows evaluations concerning the impact of different risk properties on the transfer of error to be made. For the current study, two main questions are asked: 1) what is the impact of error in risk data on overall classification outcomes; and 2) how does such error impact validity. The study found that risk tools generally have a low tolerance for error. The injection of 10 percent error into risk assessment information produced 25 to 40 percent error in classification outcomes. However, the injection of random error only minimally reduces classification validity by causing the subgroup recidivism/base rates for each category to mildly shrink towards the mean. Different risk tools and factors play a critical role in determining an instrument’s sensitivity to error. Specific risk properties such as dichotomous risk items, having fewer risk categories, risk items with lower weights, and having more risk items reduce the sensitivity of error in risk tools. A risk tool’s tolerance for error is, thereby, controlled by a confluence of factors. This dissertation facilitates a better understanding of the interplay between error in risk information and error in classification outcomes. The findings improve iii knowledge of the sensitivity of error in offender risk classification instruments. Furthermore, it explains how the sensitivity of error is aggravated or mitigated by the inclusion of different common risk device properties. iv Acknowledgements I would first like to thank my dissertation chair, Dr. Todd Clear, for his insights and guidance throughout this entire process. Second, I would like to thank Christopher Baird and his team for sharing their risk data on which the tests were conducted. Additionally, I am indebted to Dr. Robert Apel, Dr. Joel Miller, and Eric Leneskie for their helpful comments on drafts. Finally, the completion of this endeavor would not have been possible without the support of my family, thank you. v Table of Contents Abstract……………………………………………...……………………………… ii Acknowledgements…………………………………………………………………. iv Table of Contents…………………………………………………………………… v List of Tables……………………………………………………………………….. vii Chapter I-Introduction………………………………………………………………. 1 Importance…………………………………………………………………. 2 Purpose……………………………………………………………………... 5 Classification for Risk……………………………………………………… 6 Functions and Classification Characteristics………………………………. 6 Classification and Prediction………………………………………………. 11 Accuracy/Prediction Issue………………………………………………….. 15 Base-rates…………………………………………………………………… 20 Selection Ratios…………………………………………………………….. 23 Cutoff Scores……………………………………………………………….. 23 Validity in Context………………………………………………………….. 26 Constructing Classification Devices………………………………………... 27 Chapter II-Theoretical Framework………………...……………………………….. 31 Data Problems………………………………………………………………. 31 Data and Omission………………………………………………………….. 36 Transporting Risk Devices………………………………………………….. 37 Error in Application………………………………………………………… 39 Problems Focused in Study………………………………………………… 43 Chapter III-Standard for Construction and Evaluation…………………………….. 45 Informal Construction Process……………………………………………… 46 Measures of Validity……………………………………………………….. 50 Evolution and Validity……………………………………………………… 57 Reliability…………………………………………………………………… 71 Equity……………………………………………………………………….. 73 Cost and Efficiency Chicago’s Public............................................................ 75 Chapter IV-Methodology………………………...…………………………………. 78 Statement of the Problem…………………………………………………… 78 Datasets……………………………………………………………………... 79 Procedure for Injecting Error……………………………………………….. 97 Hypotheses………………………………………………………………….. 98 Analytical Plan……………………………………………………………… 105 Measurement/Statistical Analysis………………………………………...… 110 Chapter V-Assessing the sensitivity of error in risk devices- Analyses and Results.. 113 Results- Hypothesis #1……………………………………………………… 114 Hypothesis #2……………………………………………………... 119 Hypothesis #3……………………………………………………... 122 Hypothesis #4…………………………………………………….. 126 Hypothesis #5……………………………………………………... 132 Hypothesis #6…………………………………………………….. 138 Hypothesis #7……………………………………………………... 146 Hypothesis #8…………………………………………………….. 155 vi Hypothesis #9……………………………………………………… 158 Hypothesis #10…………………………………………………….. 161 Hypothesis #11…………………………………………………….. 165 Hypothesis #12…………………………………………………….. 170 Summary……………………………………………………………………. 173 Hypothesis Summary……………………………………………………….. 176 Chapter VI-Discussions and Conclusions…………………………………………... 178 Overview of Research Findings…………………………………………….. 178 Limitations………………………………………………………………….. 184 Policy/ Future Research Implications………………………………………. 188 References…………………………………………………………………………... 191 Curriculum Vitae……………………………………………………………………. 200 vii List of Tables Table 1…………………………………………………………………………….. 18 Table 2…………………………………………………………………………….. 19 Table 3…………………………………………………………………………….. 54 Table 4…………………………………………………………………………….. 56 Table 5…………………………………………………………………………….. 83 Table 6.…………………………………………………….………………………. 84 Table 7…………………………………………………….………………………. 87 Table 8…………………………………………………………………………….. 88 Table 9………………………………………………………………….…………. 91 Table 10…………………………………………………….………………………. 92 Table 11……………………………………………………….……………………. 94 Table 12…………………………………………………….………………………. 96 Table 1A………………………………………….…………………………………. 118 Table 1B…………………………………………………………………………… 118 Table 2A…………………………………………………………………………… 120 Table 2B…………………………………………………………………………… 121 Table 3A…………………………………………………………………………… 126 Table 4A…………………………………………………………………………… 129 Table 5A…………………………………….………………………………………. 135 Table 5B……………………………………….……………………………………. 135 Table 5C…………………………………………………………………………….. 136 Table 5D…………………………………………………………………………….. 136 Table 5E…………………………………………………………………………….. 137 Table 5F…………………………………………………………………………….. 137 Table 6A…………………………………………………………………………….. 140 Table 6B…………………………………………………………………………….. 140 Table 6C……………………………………………………………………………. 144 Table 7A……………………………………………………………………………. 149 Table 7B…………………………………………………………………………….. 150 Table 7C…………………………………………………………………………….. 151 Table 7D…………………………………………………………………………….. 151 Table 7E…………………………………………………………………………….. 152 Table 7F…………………………………………………………………………….. 153 Table 8A……………………………………………………………………………. 156 Table 11A…………………………………………………………………………… 167 Table 11B…………………………………………………………………………… 168 iii 1     Chapter 1 Introduction In biology, classification refers to the arrangement of living organisms into different groups due to their similarities and differences. Separating organisms into different classes or families helps scientists understand what different species have in common with one another. The classification of offenders is similar. In criminal justice, offender classification refers to the disaggregating of offenders into groups of individuals with similar attributes. Classification helps criminologists divide offenders into meaningful groups to serve a specific purpose, which will depend on the goals of agencies (Champion, 1994). Based on a scientific model, it allows offenders to be treated as members of groups for which there is an experience base (Clear, 1988). The actions of other members of the group to which they belong form this experience. Without the ability to group offenders, science in the use of classification is of no help to practitioners in structuring decisions (Clear, 1988). Misclassifications or the incorrect placement of individuals can occur if sufficient error enters to distort classification. However, initial error does not necessarily translate into final misclassification- this will greatly depend on the sensitivity of the classification instrument to error. In other words, the initial quantity of error may or may not be commensurate of the amount of error that is experienced in the final classification phase. Many factors such as base rate, distribution of data, numbers of classification categories, and number of classification items are speculated to have an impact on the sensitivity of