Technical Report # 1011 easyCBM® Mathematics Criterion Related Validity Evidence: Oregon State Test Daniel Anderson Julie Alonzo Gerald Tindal easyCBM Criterion Validity p. 2 University of Oregon Published by Behavioral Research and Teaching University of Oregon • 175 Education 5262 University of Oregon • Eugene, OR 97403-5262 Phone: 541-346-3535 • Fax: 541-346-5689 http://brt.uoregon.edu                 Note: Funds for the data set used to generate this report come from a federal grant awarded to the UO from  the Institute for Education Sciences, U.S. Department of Education: Reliability and Validity Evidence for  Progress Measures in Reading (Award # R324A100014 funded from June 2010 – June 2012) and from the  Institute for Education Sciences, U.S. Department of Education: Assessments Aligned with Grade Level Content  Standards and Scaled to Reflect Growth for Students with Disabilities (Award #R324A70188 funded from  2007‐2011).    Copyright © 2010. Behavioral Research and Teaching. All rights reserved. This publication, or parts thereof,  may not be used or reproduced in any manner without written permission.    The University of Oregon is committed to the policy that all persons shall have equal access to its programs,  facilities, and employment without regard to race, color, creed, religion, national origin, sex, age, marital easyCBM Criterion Validity p. 3 status, disability, public assistance status, veteran status, or sexual orientation. This document is available in  alternative formats upon request. easyCBM Criterion Validity p. 4 Abstract In this technical report, we present the results of a study examining the relation between the math measures available on the easyCBM® online benchmark and progress monitoring assessment system and the Oregon statewide assessment of mathematics. Designed for use within a response to intervention (RTI) framework, easyCBM® is intended to help educators identify students who may be at risk for failure. Often, students deemed at-risk are those who would not be predicted to pass the state test. In Oregon, new performance level standards were adopted for the 2010-2011 school year. We use regression and correlation analyses to examine the relation between easyCBM® and Oregon’s state test, paying particular attention to the predictive accuracy given the new performance standards. easyCBM Criterion Validity p. 5 easyCBM® Criterion-Related Validity Evidence: Oregon State Test In this technical report, we present the results of a study of the criterion validity of easyCBM® math at grades 3-8. The Oregon Assessment of Knowledge and Skills (OAKS), Oregon's state test used for accountability purposes, was used as the criterion. The OAKS was administered at the end of the school year, while easyCBM® was administered tri-annually. Thus, we examine easyCBM® for both its predictive validity, with the fall and winter measures predicting the MSP, and its concurrent validity, with the relation between the spring measure and MSP examined. The easyCBM® Progress Monitoring Assessments The online easyCBM® progress monitoring assessment system was launched in September 2006 as part of a Model Demonstration Center on Progress Monitoring funded by the Office of Special Education Programs (OSEP). At the time this technical report was published, 92,925 teachers had registered easyCBM® accounts, representing schools and districts spread across every state in the country. During the 2008-2009 school year, an average of 305 new accounts were registered each week, and the popularity of the system continues to grow. In the month of October 2010, alone, 11,885 new teachers registered for accounts. The online assessment system provides both universal screener assessments for fall, winter, and spring administration and multiple alternate forms of a variety of progress monitoring measures designed for use in K-8 school settings. As part of state funding for Response to Intervention (RTI), states need technically adequate measures for monitoring progress. Given the increasing popularity of the easyCBM® online assessment system, it is imperative that a thorough analysis of the measures’ technical adequacy be conducted and the results shared with research and practitioner communities. This easyCBM Criterion Validity p. 6 technical report addresses that need directly, providing the results of a study examining the predictive and concurrent validity evidence supporting the use of the easyCBM® assessments in mathematics in Oregon state schools. A further purpose of this study was to examine the relation of easyCBM® to OAKS given the newly adopted OAKS performance level standards. For instance, while previous studies have shown easyCBM® to be strong a strong predictor of OAKS classification (Anderson, Park, & Tindal, 2010), will this predictive utility change with the new standards? Methods Setting and Subjects Three Oregon public school districts participated in this study. Table 1 details the demographics of the sample by district and grade. The sample differed by district. In Districts 1 and 2, all students present on the day of testing were included. In District 3, easyCBM® was administered to a subset of students designed to match the population of students within the district. Measures and Analyses The easyCBM® mathematics tests used in this study contain 45 multiple-choice mathematics items, written specifically to address grade-level content knowledge and skills as described in the National Council of Teachers of Mathematics (NCTM) Focal Point Standards. These benchmark / screening assessments are designed to identify students at risk for failing to meet state content standards, giving educators the opportunity to provide additional instructional interventions targeted to students' needs. They are intended to be computer-administered in a group setting, with the students' classroom teachers supervising the test administration. For a full description of easyCBM® math, including its purpose and development, see Alonzo, Lai, and easyCBM Criterion Validity p. 7 Tindal (2009a, 2009b, 2009c), Alonzo and Tindal (2009a, 2009b), and Lai, Alonzo, and Tindal (2009a, 2009b, 2009c). The OAKS is a computer adaptive assessment administered in grades 3-8. Student scores are reported in Rasch Units – a continuous scale ranging from 0 to infinity. However, the Oregon Department of Education reports that most scores range from 150-300 (Oregon Department of Education, 2010). Data Analysis To examine the predictive and concurrent validity of easyCBM® we conducted regression and correlation analyses. Four separate regression models were conducted at each grade level. First, a full model was run, which included all easyCBM® assessments administered throughout the year. This model provided an indication of the total relation between easyCBM® and OAKS. Second, individual models were conducted for each seasonal administration. For the seasonal models, only the students’ total score for the seasonal benchmark was entered as a predictor. Correlations are reported in both the full model and the individual models. To visually represent the relation between easyCBM® and the OAKS, and to examine the impact of the newly adopted OAKS performance standards, scatterplots were produced for each seasonal administration. On each scatterplot, students’ OAKS scores are plotted along the y-axis, and their easyCBM® scores are plotted along the x-axis. The vertical lines on each plot represent the 20th and 50th percentiles of normative achievement on easyCBM®, while the horizontal lines represent the newly proposed cut scores for meeting and exceeding expectations on the OAKS. easyCBM Criterion Validity p. 8 Results The results of the regression analyses and all scatterplots are reported by grade in pages 10-68. Overall, the full model accounted for between 64% and 75% of the variance in OAKS, while the individual models accounted for between 48% and 68% of the variance in OAKS. Discussion The results of this study suggest a strong relation between easyCBM® and the OAKS. An investigation of the scatterplots indicates that the changes in the OAKS performance level standards did not affect the potential for easyCBM® to classify students as at-risk for failure accurately. The relation between the OAKS and easyCBM® was found to be the strongest in the upper grades; however, even at the lower grades the relation appeared strong. easyCBM Criterion Validity p. 9 References Alonzo, J., Lai, C., & Tindal, G. (2009a). The development of K-8 progress monitoring measures in mathematics for use with the 2% and general education populations: Grade 3 (Technical Report No. 0902). Eugene, OR: Behavioral Research and Teaching, University of Oregon. Alonzo, J., Lai, C., & Tindal, G. (2009b). The development of K-8 progress monitoring measures in mathematics for use with the 2% and general education populations: Grade 4 (Technical Report No. 0903). Eugene, OR: Behavioral Research and Teaching, University of Oregon. Alonzo, J., Lai, C., & Tindal, G. (2009c). The development of K-8 progress monitoring measures in mathematics for use with the 2% and the general education populations: Grade 2 (Technical Report No. 0920). Eugene, OR: Behavioral Research and Teaching, University of Oregon. Alonzo, J., & Tindal, G. (2009a). The development of K-8 progress monitoring measures in mathematics for use with the 2% and general education populations: Grade 1 (Technical Report No. 0919). Eugene, OR: Behavioral Research and Teaching, University of Oregon. Alonzo, J., & Tindal, G. (2009b). The development of K-8 progress monitoring measures in mathematics for use with the 2% and general education populations: Kindergarten (Technical Report No. 0921). Eugene, OR: Behavioral Research and Teaching, University of Oregon. Anderson, D., Park, B. J., & Tindal, G. (2010). An Examination of the easyCBM® Benchmark Tests and the Oregon Statewide Tests in Grades 6-8 Mathematics. Paper presented at the annual conference of the American Educational Research Association, Denver, CO. easyCBM Criterion Validity p. 10 Lai, C., Alonzo, J., & Tindal, G. (2009a). The development of K-8 progress monitoring measures in mathematics for use with the 2% and general education populations: Grade 7 (Technical Report No. 0908). Eugene, OR: Behavioral Research and Teaching, University of Oregon. Lai, C., Alonzo, J., & Tindal, G. (2009b). The development of K-8 progress monitoring measures in mathematics for use with the 2% and general education populations: Grade 8 (Technical Report No. 0904). Eugene, OR: Behavioral Research and Teaching, University of Oregon. Lai, C., Alonzo, J., & Tindal, G. (2009c). The development of K-8 progress monitoring measures in mathematics for use with the 2% and the general education populations: Grade 6 (Technical Report No. 0907). Eugene, OR: Behavioral Research and Teaching, University of Oregon. Oregon Department of Education. (2010). Assessment scoring: Frequently asked questions about scoring statewide assessments Retrieved April 21, 2010, from http://www.ode.state.or.us/apps/faqs/index.aspx?=88