Technical Report # 1001 Making the Good Even Better: Feedback from easyCBM Focus Groups, School Year 2009/2010 Julie Alonzo Gerald Tindal Cheng-Fei Lai University of Oregon Published by Behavioral Research and Teaching University of Oregon • 175 Education 5262 University of Oregon • Eugene, OR 97403-5262 Phone: 541-346-3535 • Fax: 541-346-5689 http://brt.uoregon.edu Note: Funds for the data set used to generate this report come from a contract awarded to Behavioral Research and Teaching from the Oregon Department of Education’s Statewide Longitudinal Data Systems (SLDS) Grant Copyright © 2010. Behavioral Research and Teaching. All rights reserved. This publication, or parts thereof, may not be used or reproduced in any manner without written permission. The University of Oregon is committed to the policy that all persons shall have equal access to its programs, facilities, and employment without regard to race, color, creed, religion, national origin, sex, age, marital status, disability, public assistance status, veteran status, or sexual orientation. This document is available in alternative formats upon request. Abstract This technical report provides a summary of feedback from teachers, administrators, and support personnel who used the easyCBM progress monitoring and benchmark assessment system during school year 2009/2010. Data were gathered from semi-structured focus groups conducted during the 2010 easyCBM August Institute at the University of Oregon. Results have been used to prioritize enhancements to the easyCBM system and identify additional areas of need for future development and training. easyCBM Focus Group Page 1 Making the Good Even Better: Feedback from easyCBM Focus Groups, School Year 2009/2010 The easyCBM progress monitoring and benchmark assessment system (Alonzo, Tindal, Ulmer, & Glasgow, 2006) was developed as part of a model demonstration project funded by the Office of Special Education Programs in 2006. Initially available only to a few school districts in the service area of the university where it was developed, easyCBM was first made available to a larger group of educators in the Fall of 2009. At the time this technical report was written, 24 school districts in Oregon had adopted the District version of easyCBM, joined by 8 districts in Washington state, 1 district in Mississippi, 2 in Illinois, 2 in Pennsylvania, 1 in Indiana, 1 in Ohio, 1 in North Carolina, 1 in Kansas, and 1 in Maine. As these numbers suggest, the system has received attention from a national audience and use continues to grow exponentially. As part of the developers’ ongoing effort to ensure that the system meets educators’ needs, we conducted a series of studies in the spring and summer of 2010 to gather input from teachers and administrators with easyCBM accounts. This technical report provides information from the focus group studies. Methods To capture as much information from as diverse a user group as possible, we conducted semi-structured focus groups, involving experienced users of district easyCBM and novice users the system on two separate days. Setting and Subjects The focus groups were held as part of the 2010 easyCBM August Institute that took place at the University of Oregon. The focus groups were conducted in two parts, with two different participant populations. Participants in the August 9th focus group included school and district administrators, and general and special education teachers representing five different easyCBM Focus Group Page 2 districts that had used the district version of easyCBM in School Year 2009/2010. Participants in the August 10th focus group included school and district administrators, general and special education teachers, and Educational Service District representatives from the state of Oregon, all of whom would be using the district version of easyCBM for the first time in the Fall of the 2010/2011 school year. Approximately 80% of participants in the August 10th focus group had participated as silent observers in the August 9th focus group. Measurement/Instrument Development The semi-structured focus groups were conducted as part of the 2010 easyCBM August Institute hosted by Behavioral Research and Teaching at the University of Oregon. Participants were provided with the results of two surveys conducted earlier in the spring and summer and asked to respond to questions related to their schools’ and districts’ capacity to use the computer- based easyCBM system, the student information database they used in their districts and its ability to interface with the easyCBM system, the relation between use of the easyCBM system and student performance on the Oregon statewide assessment of reading and mathematics, and the adequacy of the training available to them. During the August 9th focus group, attendees from districts that had used the district version of easyCBM during School Year 2009/2010 were asked to sit in a circle, facing one another. Attendees from districts preparing to adopt the system for initial use in School Year 2010/2011 sat in a larger circle, surrounding the first group. They were asked to jot down notes on topics they thought were most relevant during the focus group. These notes were later transcribed by a research assistant who had attended the August easyCBM Institute. The facilitator, the first author on this technical report, posed questions to the group and encouraged participation from all attendees, with follow-up questions as they arose. The third author on this technical report served as an observer and note-taker during the focus group sessions. easyCBM Focus Group Page 3 For the August 10th focus group, attendees were first provided approximately 4 hours’ training on the district version of easyCBM. Following a lunch break, they were then brought back together for the actual focus group. During their focus group time, attendees were asked to reflect on their observations from the prior day’s focus group as well as their notes from the training provided earlier that day, and to share questions and concerns related to their capacity to use the district easyCBM system. The same facilitator, note-takers and observers participated in the same roles on both days, providing consistency across the two groups. Data Preparation and Analysis Data from the survey were gathered through the online survey tool SurveyMonkey, which presents summaries of responses to each question as well as lists of all constructed- responses provided. These data are reported in full in this technical report. Data from the focus groups were compiled by the third author on this technical report and are reported in full in this technical report. Results The focus groups addressed areas of technology needs and capacity, training needs and capacity, and additional resource needs. Findings from each of these three broad categories are presented separately. Technology Needs and Capacity Focus group participants reported a variety of student information systems, with the following programs referenced: Pentamation, School Master, ESIS, and Skyward. All participants indicated that their student information system interfaced easily with the easyCBM system, allowing smooth and accurate transfer of student and staff data across the different platforms. Participants indicated strong interest in having the Oregon statewide assessment scores included in the easyCBM reports, indicating that this additional piece of data would easyCBM Focus Group Page 4 facilitate decision making. A few participants discussed a need for a more automated means of updating the student records on easyCBM, and there was some discussion about inter-district collaboration on writing the computer code that would facilitate this happening. Although participants on both days indicated that the demands for computer lab time to finish the state assessment in the spring presented challenges for finishing the spring benchmark assessments on easyCBM, they also indicated that this was not as much of a problem during the fall or winter benchmark windows. Participants’ benchmark windows ranged from a couple of days for the smallest districts to three weeks to complete the full set of benchmark assessments across their district. All participants on both days indicated that the data download features on easyCBM were accessible and easily understood. The biggest challenge, related to technology, identified by participants involved correctly linking students with adult staff using the student/teacher association table. This problem appeared to be a challenge primarily because this information did not necessarily already exist in the Student Information Systems used by the different districts. Returning district users on August 9th mentioned the importance of using the unique State Student Identification number rather than a locally-created student ID number as the system expands so that students from different districts can be passed from one district to the next without losing any of their historical easyCBM data. Training Needs and Capacity Participants on both days were provided with the PowerPoint presentation prepared by the lead author on this technical report as a tool to introduce others to the district easyCBM site. Participants in the August 9th focus group indicated that similar PowerPoint presentations that had been made available to them the prior year when they received their initial training on the easyCBM Focus Group Page 5 system had been useful to them as they provided professional development to others in their districts. Some participants reported that they found the training videos and materials available on the easyCBM site useful as they worked with others in their districts. A common theme among participants on both days was the need for ongoing professional development, not only on using the easyCBM system itself but also on instructional interventions and strategies to support students whose easyCBM performance indicated they needed additional academic support. A common theme on both days was the need for more information related to mathematics. Some participants on both days referenced the Oregon Data Project and emphasized the connections they saw between that state effort and effective use of the easyCBM system. Participants expressed support for ongoing collaborative networking between districts using easyCBM in the state. They indicated that such collaboration could help provide ‘shortcuts’ for new districts starting to use the system by providing a network of educators from across the state who might be able to assist with questions related to uploading student and staff data, interpreting the results, or recommending instructional programs, strategies, and interventions. In addition, participants indicated that involvement of Educational Service Districts (ESDs) throughout the state would help provide the infrastructure needed to expand use of the easyCBM system, particularly in the smaller and more remote districts. Representatives from three ESDs were present during both focus groups, and they added their support for this suggestion. Additional Resource Needs Participants of both focus groups shared their enthusiasm for developing a “resource and training” system where districts could place resources they had developed to assist in the use of benchmarking and progress monitoring data to guide instructional decision making. One easyCBM Focus Group Page 6 participant shared concerns about quality control, and others in attendance agreed that there would be a need for any resources to be ‘vetted’ prior to being placed on the site for dissemination to ensure that they met quality standards. During the August 10th focus group, participants discussed the potential for using a more-widely fleshed out data system that also included specific information about students’ program of study and the specific instructional strategies, curriculum, and interventions they were being provided. They thought such a data system might provide much-needed data on the effectiveness of different approaches, particularly in mathematics, where little research currently exists. Discussion Results of the focus groups provide evidence that the current district easyCBM system is widely-accessible and provides useful information to districts in Oregon. The existing technology interfaces well with a variety of student information systems in use across the state and provides easily-accessible student data appropriate for district, building, and classroom-level educators. The existing training materials available on the easyCBM system, which provide instruction on administering and scoring the benchmarking and progress monitoring assessments (training link on the easyCBM home page) and on interpreting the benchmark, group, and individual reports (teachers’ manual, downloadable PDF on the easyCBM home page), meet district needs for those particular areas. However, participants clearly identified a need for additional information on how to intervene with students once they are identified as being at risk, particularly in the area of mathematics. easyCBM Focus Group Page 7 References Alonzo, J., Tindal, G., Ulmer, K., & Glasgow, A. (2006). Easycbm assessment system, FAQ. Eugene, OR: Behavioral Research and Teaching.