Developing and Selecting Assessments of Student Growth for Use in Teacher Evaluation Systems Joan L. Herman, Margaret Heritage, and Pete Goldschmidt AACC: Assessment and Accountability Comprehensive Center: A WestEd and CRESST partnership. aacompcenter.org Copyright © 2011 The Regents of the University of California. The work reported herein was supported by WestEd, grant number 4956 s05-093, as administered by the U.S. Department of Edu- cation. The findings and opinions expressed herein are those of the author(s) and do not necessarily reflect the positions or policies of the AACC, CRESST, WestEd, or the U.S. Department of Education. Herman, J. L., Heritage, M., & Goldschmidt, P. (2011). Developing and selecting assessments of student growth for use in teacher evaluation systems. Los Angeles, CA: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST). The authors thank Tamara Lau and Karisa Peer (CRESST) for design and editorial support. Developing and Selecting Assessments of Student Growth for Use in Teacher Evaluation Systems¹ Joan L. Herman, Margaret Heritage, and Pete Goldschmidt Across the country, states and districts are grappling to support long-term plans to validate assessments. with how to incorporate assessments of student learning However, we recognize that states and districts must into their teacher evaluation systems. Sophisticated respond to current policy mandates. Thus, operating statistical models have been proposed to estimate the under both limited resources and tremendous time relative value individual teachers add to their students’ pressure, they cannot be expected to address the entire assessment performance (hence the term teacher “value- framework. Nevertheless, by understanding the basic added” measures). The strengths and limitations of these requirements the student assessments need to satisfy, statistical models, as well as the value-added measures and the design features that are central, we believe they produce, have been widely debated; yet, little that our guidance can help states and districts move attention has been devoted to the quality of the student forward, accumulating important evidence and making assessments that these models use to estimate student improvements in the quality of assessments. growth, which is fundamental to the trustworthiness of The Basic Argument Justifying Use in any teacher value-added measure. Teacher Evaluation Assessments that nominally address the subject or grade •••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••• level that educators teach do not necessarily suffice for the Validity is the overarching concept that defines quality purpose of measuring growth and calculating the value in educational measurement. In essence, validity is that teachers contribute to that growth. In fact, student the extent to which an assessment measures what it is growth scores require at least two assessments of student intended to measure and provides sound evidence for learning - one near the beginning of the school year or the specific decision-making purposes. Assessments in and of end of the prior year and another at the end of the current themselves are neither valid nor invalid. Rather, validation school year. Carefully designed and validated assessments involves evaluating or justifying a specific interpretation(s) are needed in order to provide trustworthy evidence of or use(s) of the scores. teacher quality. Herein lies the purpose of this brief: to The process of justifying the use of student growth scores provide guidance to states and districts as they develop for teacher evaluation takes the form of an evidence-based and/or select and refine assessments of student growth argument that links student performance on assessments so that the assessments can well serve teacher evaluation to specific interpretations, conclusions, or decisions that purposes. are to be made on the basis of assessment performance. Applicable across content areas and grade levels, the The argument is set out as a series of propositions and guidance is grounded in a validity framework that: attendant claims requiring substantiation with evidence. 1. Establishes the basic argument, which justifies the Propositions use of assessments to measure student growth as The general propositions that comprise the argument are: part of teacher evaluation 1. The standards clearly define what students are 2. Lays out essential claims within the argument that expected to learn. need to be justified 2. The assessment instruments are designed to 3. Suggests sources of evidence for substantiating the accurately and fairly address what students are claims expected to learn. 4. Uses accumulated evidence to evaluate and improve 3. Student assessment scores accurately and fairly score validity measure what students have learned. The framework is purposively comprehensive in laying out a broad set of claims and potential evidence intended ¹This brief is a shortened version of Guidance for Developing and Selecting Student Growth Assessments for Use in Teacher Evaluation. For those who wish to have more details about the contents herein, please refer to the extended Guidance. Developing and Selecting Assessments of Student Growth for Use in Teacher Evaluation Systems • Joan L. Herman, Margaret Heritage, and Pete Goldschmidt 3 4. Student assessment scores accurately and fairly of student growth; whereas, the third and fourth measure student growth. propositions target psychometric and technical qualities of student scores. The final proposition focuses on the 5. Students’ growth scores (based on the technical quality of the teacher value-added scores, which assessments) can be accurately and fairly are generated from the individual student growth scores attributed to the contributions of individual using complex statistical models. While some would teachers. regard this final proposition as beyond the province of test Although the first proposition clearly falls outside of validation, we include it as an essential part of the validity the domain of test development and validation, it is an argument and the ultimate link between the test scores to essential requisite for it. Assessment development and/ their intended evaluation use. or selection for purposes of teacher evaluation must Figure 1 displays these propositions as a series of if/then be guided by publically available and agreed upon statements, which comprise the argument justifying that learning expectations and not simply by what is easy or student assessments can be used to measure student convenient to test. growth for the purpose of evaluating teachers. The The second general proposition highlights the sequence of propositions represents the successive issues importance of sound instrument design, development, that states and districts should attend to as they select, and review processes in creating trustworthy measures develop, and/or refine measures of student growth to evaluate teachers. Standards clearly define The assessment instruments have been designed to If learning expectations and if yield scores that can accurately and fairly reflect student: for the subject area and 1. achievement of the standards each grade level 2. learning growth over the course of the year There is evidence that student There is evidence that growth scores accurately and and if the assessment scores and if fairly measure student progress accurately and fairly measure over the course of the year the learning expectations There is evidence Interpretation of scores and if that assessment scores then may be appropriately used represent teachers’ to inform judgments about contribution to student growth teacher effectiveness Adaptation based on Bailey and Heritage, 2010; Perie and Forte (in press). Figure 1. Propositions that justify the use of these measures for evaluating teacher effectiveness. Essential Claims and Evidence claims, at least in part, can be examined a priori through •••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••• evidence produced by rigorous expert review. With the propositions laid out, the next step in Psychometric and other technical quality claims. validation involves establishing claims and evidence Claims about the technical quality of the scores and sources that are important for evaluating each how well they function as measures of student learning proposition (see Table 1). Like the propositions, and of teachers’ contributions to student progress. The claims are of two basic types: 1) design claims and evaluation of these claims draws largely on student 2) psychometric and other technical quality claims. data from large-scale field tests or, if necessary, from Design claims. Claims about the attributes and operational administrations of the assessments and on characteristics of the assessment instrument and item special research studies that can be coordinated with design that are likely to yield sound measures. These field testing and administration. Developed to support the work of the Assessment and Accountability Comprehensive Center • Spring 2011 4 Table 1. Propositions and Claims Critical to the Validity Evaluation. Proposition 1 - Standards clearly define learning expectations for the subject area and each grade level Design Claims: Evidence • Learning expectations are clear • Expert reviews • Learning expectations are realistic • Learning expectations reflect a progression (at minimum for the span of a grade level) Proposition 2a - The assessment instruments have been designed to yield scores that can accurately and fairly reflect student achievement of the standards Design Claims: Evidence • Specifications/blueprint for assessment reflect the breadth and depth of • Expert reviews of alignment learning expectations • Measurement review of administration and • Assessment items and tasks are consistent with the specifications and scoring procedures comprehensively reflect learning expectations • Sensitivity reviews • Assessment design, administration, and scoring procedures are likely to produce reliable results • Assessment tasks and items are designed to be accessible and fair for all students Proposition 2b - The assessment instruments have been designed to yield scores that can accurately and fairly reflect student learning growth over the course of the year Design Claims: Evidence • Assessments are designed to accurately measure the growth of individual • Expert reviews students from the start to the end of the school year • Research studies • Cut scores for defining proficiency levels and adequate progress, if relevant, are justifiable • Assessments are designed to be sensitive to instruction Proposition 3 - There is evidence that the assessment scores accurately and fairly measure the learning expectations Psychometric Claims: Evidence • Psychometric analyses are consistent with/confirm the assessment’s learning • Psychometric analyses specifications/blueprint • Content analysis • Scores are sufficiently precise and reliable • Scores are fair/unbiased Proposition 4 - There is evidence that student growth scores accurately and fairly measure student progress over the course of the year Psychometric Claims: Evidence • Score scale reflects the full distribution of where students may start and • Psychometric modeling and fit statistics end the year • Sensitivity/bias analyses • Growth scores are sufficiently precise and reliable for all students • Growth scores are fair/relatively free of bias • Cut points for adequate student progress are justified Proposition 5 - There is evidence that scores represent individual teachers’ contribution to student growth Psychometric Claims: Evidence • Scores are instructionally sensitive • Research studies on instructional sensitivity • Scores representing teacher contribution are sufficiently precise and reliable • Precision and stability metrics • Scores representing teachers contributions are relatively free of bias • Advanced statistical tests of modeling alternatives and tenabiliity of assumptions Based on Herman & Choi, 2010 Developing and Selecting Assessments of Student Growth for Use in Teacher Evaluation Systems • Joan L. Herman, Margaret Heritage, and Pete Goldschmidt 5 “ “ Expert review. Note in Table 1 that expert review is called for in evaluating claims for Propositions 1 and Even for experts, 2. Highly qualified individuals should comprise review panels—including experts in subject matter, instruction it is difficult to and learning, English learners, students with disabilities, culturally diverse students, measurement and assessment, ascertain what an as well as expert teachers. Their reviews do not all have to be conducted serially; instead, expert panels can convene assessment or item to conduct reviews simultaneously for many of the design claims for each of the propositions. measures simply by The expert panel should engage in structured ratings of the claims, such as those devised by Norman Webb and looking at it. Andrew Porter. The ratings can be analyzed to provide empirical indices of how well the specifications and actual assessments align with target standards, the frequency of potential bias, and sensitivity or reliability problems. Ratings can also be utilized to summarize what is good, Problems at an early stage portend larger ones bad, and missing in needed rubrics, administration, subsequently. Required psychometric and statistical training, and scoring procedures. Furthermore, it is often models become increasingly complex as one moves useful to examine the extent of expert agreement: high through the continuum from evaluating individual agreement increases confidence in findings; whereas, assessments to evaluating the accuracy and fairness of low agreement may be cause for concern. The expert teacher value-added scores. It is likely that states and reviews provide important feedback that can either be districts will need to consult measurement and statistical used immediately to strengthen identified weaknesses, experts to conduct the analyses and review the results or if time is limited, be used in future years to improve (e.g., analysis and review of the specific models used, assessment quality. and the meaningfulness and robustness of estimates with regard to reliability, precision, and stability data). Be aware, however, that expert review has its limits. Even for experts, it is difficult to ascertain what an assessment Reciprocal relationships. Although we have or item measures simply by looking at it. For performance differentiated design and psychometric claims (and the assessments or expensive assessments in particular—time evidence on which each is based), it is important to permitting—it is worthwhile to do small scale think- note the reciprocal relationships between them. On the aloud or cognitive lab studies. These studies ask students one hand, the design claims provide the foundation for to think aloud as they respond to select items or tasks. the technical qualities referred to in the psychometric Student responses are then analyzed to determine whether claims. On the other hand, the evidence related to the the tasks actually elicit the content and cognitive demands psychometric and technical quality claims can identify that were intended, and/or whether the tasks include assessment weaknesses that need further refinement or unintended obstacles preventing some students from may raise issues for additional study. showing their knowledge and skills. At the same time, the two kinds of evidence are Psychometric evidence. As attention moves from frequently used in concert to identify and respond to design claims to psychometric claims, the demands potential challenges in the meaning and comparability for specialized measurement and statistical knowledge of assessment scores. Fairness, for instance, is always a progressively increase. central concern in assessment. Applying Universal Design principles during the design phase means that assessment The sequence of propositions suggests that the development takes the characteristics of all students for psychometric claims first focus on the individual whom the assessment is intended to take into account assessments, which will be used to comprise student (e.g., English learners, students with disabilities, culturally growth scores (e.g., assessments given at the beginning diverse students) and helps assure that items and tasks and at the end of the academic year). Next, the focus will be accessible to as many students as possible. Items moves to the growth scores and to the teacher value- and tasks also are routinely subjected to sensitivity added measures that are derived from the student growth reviews prior to field-testing or operation use. Even so, scores. Developed to support the work of the Assessment and Accountability Comprehensive Center • Spring 2011 6 psychometric analyses may well uncover some items and/ or development processes; moreover, they can use that appear problematic or function differently for the full set to establish a continuing validation agenda. students from different subgroups. These items will As the sequence of propositions indicates—states need to be re-examined by relevant experts to determine and districts should start by establishing clarity on whether a bias problem exists and, if so, whether to learning expectations and ensuring, as best they can, eliminate or remedy it. that selected or developed assessments are well-aligned with those expectations and do not contain fatal design Accumulated Evidence to Evaluate Validity flaws. If necessary, evidence for evaluating subsequent •••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••• propositions can be collected and analyzed in concert Validity is a matter of degree (based on the extent to with the assessments’ first and subsequent operational which an evidence-based argument justifies the use administrations. of an assessment for a specific purpose). A complete For instance, states and districts can use the design validity argument, supporting the interpretation and use claims and evidence from expert reviews—along with of growth assessments to evaluate teacher effectiveness, any available technical data related to the psychometric would appraise all of the claims and diverse evidence claims—to systematically evaluate and select the best sources listed in Table 1. available options from existing assessments. They Whether based on a full argument or only on selected can use this evaluation, especially the strengths and claims for which data are available, the appraisal is weaknesses it identifies, to refine the assessment. Over likely to show areas of strength and weakness and time, additional evidence can be collected to evaluate suggest areas where assessments may be strengthened to a fuller set of claims and used, if needed, to further better serve teacher evaluation purposes. The appraisal improve the measures. Just as educators are expected may also raise issues where additional evidence is to use evidence of student learning to improve their needed. Validation, in short, is an iterative process practice, so too should we expect states and districts that serves both to build the case for the use of the to utilize evidence of validity to improve their use of assessment and support improvements in assessment student growth measures for teacher evaluation. design, interpretation, analysis, and use. Finally, we underscore that no assessment, including student growth assessment, is free of error and all are Conclusion imperfect. The Standards for Educational and Psychological •••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••• Testing (1999) highlights that no important decision This brief has identified an extensive set of propositions, should be based on the results of a single assessment claims, and evidence sources that are important to the because one evaluation cannot adequately capture the validity argument and which justify the use of student multi-faceted domain of teacher effectiveness; therefore, growth assessments as part of teacher evaluation. As we multiple measures are essential. Assessments of student indicated earlier, the set is aspirational; hence, we expect growth must be as good as possible; yet, we must keep the validity argument to unfold over time. in mind that they are only one part of a sound teacher Under strong policy mandates, many states and districts evaluation system. had to adopt aggressive timelines for implementing teacher evaluation systems that incorporate student growth as a component for all grades and subjects. This rapid press for implementation means that it is unlikely that the student growth measures used in the early stages of an evaluation system’s implementation will meet all (or even many) of the criteria laid out in this brief. Nonetheless, we hope that this guidance will aid states and districts to reflect on the major areas of concern as well as initiate a long-term, systematic process to develop relevant evidence, evaluate strengths and weaknesses, and improve the assessments they adopt. States and districts can utilize the initial propositions and attendant claims to guide their assessment selection Developing and Selecting Assessments of Student Growth for Use in Teacher Evaluation Systems • Joan L. Herman, Margaret Heritage, and Pete Goldschmidt 7 References Bailey, A., & Heritage, M. (2010). Washington state English language proficiency assessment foundations document. Evaluating the Validity of English Language Proficiency Assessments Project (EVEA; CFDA 84.368). Herman, J. L., & Choi, K. (2010). Validation plans for Gates-funded assessments English-language arts and mathematics. Los Angeles, CA: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST). Perie, M., & Forte, E. (in press). Developing a validity argument for assessments of students in the margins. In M. Russell (Ed.), Assessing students in the margins: Challenges, strategies, and techniques. Charlotte, NC: Information Age Publishing. Porter, A.C. (2002). Measuring the content of instruction: Uses in research and practice, Educational Researcher, 31 (7), 3–14. American Educational Research Association (AERA), American Psychological Association (APA), and National Council for Measurement in Education (NCME). (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association. Webb, N. L. (1997). Criteria for alignment of expectations and assessments in mathematics and science education. (Research Monograph No. 6). Madison, WI: University of Wisconsin-Madison, National Institute for Science Education. Webb, N. L. (2002, December). Alignment study in language arts, mathematics, science, and social studies of state standards and assessments for four states. Washington, DC: Council of Chief State School Officers. Developed to support the work of the Assessment and Accountability Comprehensive Center • Spring 2011 8