DOCUMENT RESUME ED 136 425 EA 009 349 AUTHOR Owens, Thomas R.; Hiscox, Michael D. Alternative Models for Adversary Evaluation: TITLE Variations on a Theme. Apr 77 PUB DATE NOTE 29p.; Paper presented at the Annual Meeting of the American Educational Research Association (New York, N.Y., April 4-8 i977); Pages 23-24 may not reproduce clearly due to small print size EDRS PRICE MF-$0.83 HC-$2.06 Plus Postage. DESCRIPTORS Case Studies (Education); Elementary Secondary Education; *Evaluation Methods; Higher Education; Models IDENTIFIERS *Adversary Evaluation ABSTRACT Growing dissatisfaction with conventional evaluation approaches to dealing with complex social phenomena has led evaluators to search out alternative evaluation models. Exclusive reliance on the experimental model and the logic of statistical inference has been seriously attacked on philosophic, technical and pragmatic grounds. This paper deals with existing problems in traditional evaluation and presents a rationale for exploring alternative approaches, along with a brief case study description of six applications of adversary evaluation using the judicial and debate forms. Some common features noted in the six adversary evaluation case studies were 1) better communication between evaluators and decision makers, 2) greater attention to the formulation of key evaluation issues, and 3) increased concern for meta-evaluation. The authors conclude with seven unresolved issues in adversary evaluation. (Author) *********************************************************************** Documents acquired by ERIC include many informal unpublished * materials not available from other sources. ERIC makes every effort * * to obtain the best copy available. Nevertheless, items of marginal * * reproducibility are often encountered and this affects the quality * * of the microfiche and hardcopy reproductions ERIC makes available * * via the ERIC Document Reproduction Service (EDRS). EDRS is not * responsible for the quality of the original document. Reproductions * * supplied by EDRS are the best that can be made from the original. *********************************************************************** ALTERNATIVE MODELS FOR ADVERSARY EVALUATION: VARIATIONS ON A THEME U S. DEPARTMENT OF HEALTH. EDUCATION &WELFARE NATIONAL INSTITUTE OF EDUCATION THIS DOCUMENT HAS BEEN REPRO- DUCED EXACTLY AS RECEIVED FROM THE PERSON OR ORGANIZATION ORIGIN- ATING IT POINTS CF VIEW OR OPINIONS STATED DO NOT NECESSARILY REPRE- SENT OFFICIAL NATIONAL INSTITUTE OF EDUCATION POSITION OR POLICY Thomas R. Owens and Michael D. Hiscox Northwest Regional Educational Laboratory 710 S.W. Second Avenue Portland, Oregon 97204 Paper Presented at the American Educational Research Association Annual Meeting in New York City, New York, April 1977 a ALTERNATIVE MODELS FOR ADVERSARY EVALUATION: VARIATIONS ON A THEME Thomas R. Owens and Michael D. Hiscox Northwest Regional Educational Laboratory, Portland, Oregon Growing dissatisfaction with conventional evaluation appkoaches to dealing with complex social phenomena has led evaluators to search out alternative evaluation models. Exclusive reliance on the experimental model and the logic of statistical inference has been seriously attacked on philo- sophic, technical, and pragmatic grounds. This paper deals with existing problems in traditional evaluation and presents a rationale for exploring alternative approaches, along with a brief history and description of some adversary models of evaluation, some common features of alternative approaches, and some unresolved issues in adversary evaluation. Why all the Fuss? In struggling with the problems of providing useful information to decision makers and making reasonable judgments about the merits of complex educational and social programs, evaluators have become increasingly aware of the limitations of experimental evaluation designs. As Levine (1974) has said, The experiment cannot deal with historical contexts, and it requires the reduction of whole human events to be contrived dimensions that can be quantified. Moreover, the transactional nature of human inter- action precludes the assumptions that experiments make about humans as interchangeable, reactive creatures who each respond to the stimulus and the meaning for it the experimenter had in mind. 3 In the search for methodological purity, social scientists have often lost sight of the substantive problems the methods were meant to solve. (p. 674) House (1976, p. 1) expresses dissatisfaction with reliance on the traditional "student-gain-by-testing" approach on philosophic grounds. He believes that such an approach is based on utilitarian ethics which aim to "achieve the greatest net balance of satisfaction as summed over all individuals" using a common index of satisfaction (generally, standardized test scores). The utilitarian ethic ignores the need to consider multiple outcomes and measures, emphasizes the magnitude of test score gains without consideration for processes used to achieve them, ignores the pluralistic values of diverse groups interested in the evaluation, and often favors the growth of the higher social classes at the expense of the lower. On technical grounds, "the logic of experimentation and of statistical inference demands conditions that are difficult, if not impossible, to meet in human studies." (Levine, 1974, p. 663) Problems include the use of comparison group data, the presence of extraneous variables, experimenter effects, the limitations of existing measurement instruments, and the generalization of findings to new settings. From a pragmatic point of view, many evaluation studies simply end up on someone's shelf because they are unresponsive to the needs of people involved in or affected by a program being evaluated. Wolf (1975) states that This occurs because most of the current methods are rooted in behavioral and social science research and rely on quantification and technical analysis. Great collections of numbers, such as those found in children's 4 2 cumulative files and school or program evaluation studies, tend to blur and obscure rather than sharpen and illumi- nate the education process. In seeking objectivity, the decision maker using these methodologies may exclude a factor that ought to be of fundamental concern: human judgment. (p. 185) What's Involved in Adversary Evaluation? Dissatisfaction with traditional evaluation processes in education has led to the exploration of new approaches borrowed and adapted from such fields as accounting, anthropology, economics, law and politics. This paper discusses recent developments in the use of adversary models in educational evaluation. These models borrow freely from legal procedures used in jury trials and admin- istrative hearings and from principles of public debate. Specific applications Before of adversary evaluations are described in the next section of this paper. considering alternative approaches, however, it may be helpful to review some common assumptions supporting these approaches. The following seven assumptions appear directly related to the adversary models of evaluation described in this paper. The fundamental evaluation instrument is the human intelligence 1. deciphering complex data. The evaluator is not a purely "rationale and impartial spectator," 2. but is subject to certain biases that require controls. Social and educational phenomena are multidimensional; therefore, 3. an effective e,;aluation must be responsive to these various dimen- Qualitative as well as quantitative data are needed. sions. Evaluation occurs in a pluralistic society in which differing 4. value persoectives must be addressed. 3 Decision makers are interested in considering alternative 5. interpretations of evaluation data. An adversary approach to evaluation can uncover some signifi- 6. cant insights into phenomena that are important for decision making but would likely be overlooked in a traditional study by an "objective" evaluator. Important decisions regarding large scale programs are 7. seldom made by a single individual. Therefore, an effective evaluation needs to incorporate a wide range of input from various people and to communicate the findings and interpre- tations to a broad audience. What Alternative Approaches to Adversary Evaluation Have Been Applied? This section provides case study summaries of six adversary evaluations. In working with the model, we have become aware of several uses for adversary In addition, we surveyed eleven other evaluators throughout the methods. It is probable, neverthe- country for information on additional case studies. less, that we have overlooked other good examples of adversary evaluations. A Course of Study Man: 1. The earliest reported application of an adversary approach to evaluation occurred in the form of an experimental hearing held at the Hawaii Curriculum The purpose of the experi- Center at the University of Hawaii in February 1970. ment was to examine the usefulness of a modified judicial model--the adminis- The focus was trative adversary hearing--as an aid to curriculum evaluation. on exploring the technique rather than...the substantive The hear- concern, so a hypothetical issue was used. A Course ing was to decide whether the curriculum Man: 6 4 of Study, developed by the Educational Development Center in MassachuSets, should be adopted for the public schools in Hawaii. (Owens, 1973, p. 300) Two educators with experience in teaching social science served as advocate and adversary regarding potential curriculum adoption. In preparing their cases, both selected and interviewed witnesses and studied existing docu- ments related to this curriculum and the Department of Education's social studies guide. The hearings officer organized the pretrial confer- ences with the defense and prosecution, established flexible rules of operation, required that a written outline of the arguments against the adoption of the curriculum be given to the defense and to himself 4,- before the hearing, conducte,4 the hearing, ruled on the admissibility ofevidence and the propriety and relevance of questions if objectives were raised by either party, limited the hearing to two hours, and specified which points were in contention at any time during the hearing. (p. 301) A representative jury consisted of a public school sixth grade social studies teacher, the State Department of Education's program specialist in social studies and a university evaluation officer. Anproximately 40 members of the university faculty and the State Depart- ment of Education accepted an open invitation to observe and comment on the Their consensus was that the two-hour hearing served as an excellent hearing. Nevertheless, the majority way of clarifying many facets of a new curriculum. felt that two hours was insufficient time in which to reach a verdict. 7 5 Experience-Based Career Education 2. In the summer of 1974, a prototype adversary hearing was conducted by the Career Education Program of the Northwest Regional Educational Laboratory (NWREL) in Portland, Oregon. At issue was the question of whether Experience-Based Career Education (EBCE) should be adopted by school districts in the Pacific Northwest. EBCE is a new approach to secondary education, developed by the National Institute of Education and four regional educational laboratories to help bridge the gap between the classror' EBCE students rid the'community. learn basic skills, life skills and career development skills through direct experience with adults in the community. As development of the EBCE program stabilized and the program moved into a replication and dissemination phase, the need grew for an evaluation mech- anism that would help potential adopters recognize the strengths and weaknesses It was felt that a traditional evaluation report would not of the program. present enough balanced information on those issues important to potential Therefore, the decision was made'to use an adversary approach. adopters. A videotape of this adversary hearing was to be to aid a school district in deciding whether to adopt EBCE. But the hearing was intended as only a starting point in refining the application of an adversary approach to Many procedures of a complete hearing were not followed educational evaluation. precisely because of limited resources or lack of understanding regarding how For examole, the chaxges were not clearly specified, rules of to apply them. evidence were not fully developed, only four witnesses were used and the processes of jury participation and deliveration were omitted. Nevertheless, certain features of the judicial process (such as preliminary rules of evidence, pretrial discovery, interviewing of witnesse-1 and examination of evidence, use of a hearings officer, testimony and cross-examination of witnesses, and initial 8 6 and concluding statements of the advocate and adversary) were employed and adopted to program evaluation. As a result of a planning conference, a set of guidelines was drafted for use by the project coordinators in producing the hearing videotape. Over the next few weeks, these guidelines evolved into the actual specifications for production of the trial. The most important of these specifications are noted here. After introductory presentations about EBCE, several potential 1. adopter groups of educators were asked to list special strengths and weaknesses of the EBCE program. The most frequent responses were then put on.a questionnaire that was completed by a differ- ent group of potential adopters. On this questionnaire specific . issues concerning EBCE were stated, and respondents were instructed to identify, on a five-point rating scale, the extent to which they felt each issue was a strength or a weakness of EBCE. Based on this feedback, the evaluators identified the five principal positive and the five principal negative points of contention for the For each of these ten points of contention, project per- trial. sonnel identified witnesses, possible lines of testimony and specific supporting or rejecting evidence for use by the advocate and adversary. The advocate and adversary, both educators, were selected on the 2. basis of interest in the adversary method, independence from the E3CE project, ability to communicate effectively, and-necessary skill to quickly and accurately evaluate the essential aspects of an z.ducational program. Dr. Wolf of Indiana University and Dr. Terry Denny of the University of Illinois were chosen. 9 7 A local law school professor was selected as hearings officer. 3. His input to the overall production was valuable but minor because of his limited role (mainly that of presiding over the courtroom trial). However, he did provide a very helpful critique of the proposed legal techniques. A great deal cif effort over a four-month period went into 4. selecting the most important trial issues, verifying the selec- tion with school administrators, and formulating arguments and collecting evidence. The total information accumulated was probably sufficient for a trial of several days. However, this information was used sparingly because of our decision tc pro- duce only a one-hour videotape of the prototype hearing. Coordinators developed guidelines which covered rules of evidence, 5. cross-examination techniques, and so forth. Though their work provided an interesting opportunity to explore the intricacies of the legal model, the prototype hearing produced was not sufficiently complex for these guidelines to have any noticeable impact. As the trial date approached, it became obvious that a one-hour 6. trial would not allow time for sufficient discussion of issues, nor would it permit more than a superficial deliberation of the adoption question. Therefore, plans for a jury deliberation were It was also decided that some prior knowledge, gained dropped. from brochures, site visits or project orientation sessions, would be required of potential EBCE adopters before watching the video- tape would be beneficial. 10 8