DOCUMENT RESUME IR 017 707 ED 392 416 Kelly-Benjamin, Kathleen AUTHOR Teachers, Technology & Testing: Exploring New Tools TITLE for Alternative Assessment. National Science Foundation, Arlington, VA. SPONS AGENCY PUB DATE 95 ESI-9255790 CONTRACT 6p.; In: "Emerging Technologies, Lifelong Learning, NOTE NECC '95"; see IR 017 705. Speeches/Conference Descriptive (141) Reports PUB TYPE Papers (150) MF01/PC01 Plus Postage. EDRS PRICE Academic Achievement; Computer Uses in Education; DESCRIPTORS Data Collection; *Educational Assessment; *Educational Technology; Elementary Secondary Education; Measures (Individuals); *Student Evaluation; Technological Advancemert *Alternative Assessment; *Role of Technology; Video IDENTIFIERS Cameras ABSTRACT Instructional technology can help teachers formalize alternative methods of assessment. It allows them to record and report on student performance more fully and more dynamically. This paper discusses tools and methodologies used by a group called Teachers Using Technology to Measure Mathematics Meaningfully as they expanded their repertoire of classroom assessment techniques to incorporate instruments such as computers, video cameras, and Sunburst's Learner Profile. These tools assist not just with the delivery of tasks but with the collection of data and observations about student response. The paper also chronicles the formation of Teachers Using Technology to Measure Mathematics Meaningfully, discusses their goals and research activities, and suggests some future implications for the use of technology in alternative assessment. (Cont,ins 14 references.) (Author/BEW) *********************************************************************** Reproductions supplied by EDRS are the best that can be made from the original document. *********************************************************************** 4I U S DEPARTMENT OF EDUCATION Office of EOuCatortal Research one impros.--,ert EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) as 0 This document has been reproduced V..) received from the person Or organization v--r ct originating it 0 Minor changes have been made to improve reproduction quality CN7 C\ fr1 this Points ot view or opinions stated in CI document do not necessarily represent official OERI position or policy ILI I I II .` Teachers, Technology & Testing: Exploring New Tools for Alternative Assessment by Kathleen Kelly-Benjamin Paper presented at the NECC '95, the Annual National Educational Computing Conference (16th, Baltimore, MD, June 17-19, 1995. "PERMISSION TO REPRODUCE THIS MATERIAL HAS BEEN GRANTED BY Done lla Ingham 4, TO THE EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC)." BEST COPY AVAILABLE paper Teachers, Technology & Testing: Exploring New Tools for Alternative Assessment Kathleen Kelly-Benjamin, Ph.D. Florida Institute of Technology Science and Mathematics Education Department 150 West University Blvd. Melbourne, Fl 32901 (407) 768-8000 ext. 8126 [email protected] Key words: alternative assessment, testing, evaluation, mathematics education Abstract Since January, 1994, 15 teachers from the Teachers Using Technology to Measure Mathematics Meaningfully ("I'M)' Project have been developing assessment materials for elementary and middle school mathematics, that make use of technol- methodologies ogy for the delivery and collection of student achievement information. This paper discusses the tools and utilized by the TN' teachers as they expanded their repertoire of classroom assessment techniques to incorporate a range of technologies. The tools included computers and software, video cameras, the Newton, scanners, and the Learner Profile. Assessments methods included interviews, portfolios; and scoring rubrics. Discussion of the results of the T2M3 Project suggest ways that technology can allow teachers to embrace alternative assessments. Implications for expanding the use of technology beyond instruction to assessment offer new areas of research. This paper addresses the implications for the use of technology for alternative assessment. Introduction Teachers "test" their students using an assortment of teacher made paper and pencil exams that focus primarily on the content domain they teach. However, their assessments often go beyond these measures to include observations, interviews, continuous instructional monitoring, and performance samples of their students' work in a variety of situations. These data provide additional fvedback about individual student's growth and change and the effectiveness of the teachers' instruction. The resulting evaluations are holistic appraisals depicting broad pictures of students as learners (Brophy & Good, 1986; Carpenter, 1989; Natriello, 1987; Reynolds, 1992). These new models of assessment allow for increased understanding of how learners construct new knowledge (Cobb, 1990Greeno, 1989). However, these new assessment techniques require new tools. Instructional technology can help teachers formalize these alternative assessment. Educational technology holds promise for addressing some of the increased demands that meaningful and dynamic assessment place on teachers. For example, projects such as the Jasper Series (Vanderbilt University, 1990) highlight the possibilities of using technology in more support- ive ways. These projects have helped move educational technology from being a tool for varying traditional instructional activities to creating powerful educational environments (Scott, Cole, & Engel, 1992). However, for the most part, even teachers who are using technology for instruction have not fully incorporated these tools into their assessment repertoire (Ginsberg, Sebastian, Underwood, Anderson, Kridel, & Stevenson, 1994). Technological tools furnish the means for presenting a variety of assessment tasks and situations. Technology also allows us to record students' performance, tracking the actions they take while forming, testing, and verifying hypotheses. Lastly, technology allows for fuller reporting of information, for example, video vignettes of student performance. Affordable videotape and videodisk equipment, as well as microcomputers, calculators, and other interactive systems, can facilitate dynamic assessment (Kaput, 1992). Technology makes it feasible to capture and store students' actions and procedures and provide structured and more accurate records of prior actions. Yet, only 12% of video-based courses integrate any assessment tools as part of the courseware (Barrett, 1990). Exactly how current and future technology can contribute to assessment is still mostly speculative. Potentially, educa- tional technology promises the delivery of more realistic assessment situations and the tools for going beyond paper and National Educational Computing Conference, 1995 Page 48 3 pencil records and informal observations to more fully record student achievement. However much of the previous research related to educational technology and assessment has focused on computer-based techniques for administering, scoring, reporting, and interpreting test items (Bunderson, Inouye, & Olsen, 1989) Research examining teachers' use of educational technology for assessment is scarce. Many questions remain. The role technology takes in teachers' development and use of instructional assessment is relatively unexplored. Teachers are often excluded from the investigative process. If we are to successfully build integrated instructional and assessment systems that everyday functions of are valid and useful for teachers, then teachers must be partners in the process. They can articulate the assessment, and help .o identify specific problem sets that integrate assessment with curriculum and instruction. Their experiences in the classroom make them useful guides in the journey to reform. The entire Teachers Using Technology to Measure Mathematics Meaningfully Project (FM') had three underlying goals: (1) to determine expert and novice teachers' capabilities to develop comprehensive integrated assessment materials in math- ematics; (2) to investigate the role that technology plays in helping teachers present integrated assessment situations and record and analyze student achievement; and (3) to investigate the changes in teachers' pedagogical reasoning, pedagogical content, and schemata as they develop and use integrated instructional assessment tasks. We hypothesized that investigation of these three research areas would increase our understanding of how capable teachers are of integrating assessment with instruction. We proposed to identify some of the ways that researchers, technology experts, and assessment experts can contribute to teachers' efforts to create integrated assessment. Data related to all of the research questions were collected by interviews with the participating teachers, observations of the teachers as they developed and used assessment materials, and comparisons of teachers' field notes and actual assessment materials. Details of the entire study are presently being compiled. Final analysis of the project will be available in forthcoming articles. However, throughout the project some themes have emerged. This paper deals specifically with one of these themes: the features of available technology that enhance teachers' abilities to deliver instructional assessment to students and continuously record, analyze, and evaluate information about students' knowledge, cognitive processes, and development in mathematics. Methods Participants. Subjects for T2M3 were 15 teachers who met the following criteria: (1) taught elementary or middle school mathematics; (2) used some technology for instruction, (3) showed potential for extended involvement in the development and field testing of the -FM tasks; and (4) represented diverse groups and/or classroom situations that included students from under-repre- sented groups. At the time they were selected, three taught primary level, six taught upper elementary level, three taught middle/junior high, two taught gifted children in grades one to six, and one taught math and computer for grades five to eight. The selected teachers represented a broad range of expertise in teaching mathematics and in using technology. In addition, their schools provided very diverse populations. This teacher and student population allowed us to compare the application of tools and techniques across a wide domain of situations. Treatment The study consisted of four phases. During Phase 1, the 15 teachers were selected. Each teacher proposed a mathematics curriculum module they wished to expand to include assessment. Upon acceptance into the project each received a video the teachers, available to them camera to use in assessing themselves and their students. The camera was under the control of whenever they wished to use it. Phase 2 entailed the development and presentation of the instructional treatment to help the teachers create an assess- ment module to the math module. The training program informed the teachers about various technological methods of presenting assessment information; available technologies for evaluating learning in mathematics; models of interactive assessment in mathematics; National Council of Teachers of Mathematics (NCTM) Curriculum, Professional, and Assessment Standards; and techniques for being a teacher-researcher. Workshops, hands-on activities, and collaborative work times provided the teachers opportunities to work with content area specialists, assessment experts, cognitive scientists, and technology specialists. The teachers participating in the PM" Project received extended training in creating and using integrated mathematics modules. This assessment. Following the training, the teachers developed the instructional assessment component for their schedule of training and meetings ensured, as much as possible, that teachers received the support needed to successfully create effective assessment tasks. In order to document teachers' capabilities to create meaningful classroom assessment materials, the progress of the participating teachers was carefully observed. These observations, conducted through videotapes and in-class visits, provided of teachers who have a range of pedagogical a thorough description of factors that enhanced or impeded the successes expertise. Providing detailed explanations of the progress of the various teachers furthered our understanding of teachers' of the capacity for developing original assessment procedures. Teachers' work during the project sessions provided exemplars models for assessment. Results The collaborative investigation of the project participants provided an opportunity to articulate technology's role in classroom assessment. The results are reported in two areas: tools and methodologies. Page 49 NECC '95, Baltimme, MD 4 BEST COPY AVAILABLE Tools. An important component of the FM' was the actual technology that was available, or was made available to the teachers throughout the project. In order to depict the influence of these tools on the teachers' assessment techniques, we report our findings within three categories: school resources, new toys, and prototypes. These categorizations allowed us to represent all the tools that the teachers used during the project. School Resources. There was a range of technology available at each of the participant schools at the beginning of the project. Every school had televisions, video cassette recorders, calculators, and computers, and at least one videodisk player. Many of the 15 teachers had some of this technology in their classrooms. Specifically, all had televisions available and could procure a VCR, videodisk, or video camera. At the beginning of the project, 5 of the 15 had one computer in their classrooms (either Apple IIEs or Macintosh LCs), 4 had two or more computers (a combination of Apples and either Macintosh LCs or MS-DOS 286 or 386s), and 6 had no computers in their classrooms. By the end of the project, all but 2 of the 15 had computers in their classroom, and most had acquired newer model Macintosh or DOS computers. When the project began, only the four teachei, with more than one classroom computer were using them for anything besides extracurricular activities or rewards for students. They used their computers for student projects and for classroom management and presentation. Few were using them for any aspect of assessment except record keeping. Throughout the project relevant use increased. The TM teachers increased their use of technology to administer assessment information and record student responses. New Toys. Once into the FM' network, many of the teachers used their membership to learn about new sources of funding for equipment, to find out who had access to the resources, and to discover how to ask for their share. For many, it was also an opportunity to make better use of the technology at their schools. For example, few had used the videodisk player before the project began. Either they didn't know how to use it or didn't know what disks were available that would be appropriate to their curriculum. After being introduced to the Jasper Series many found ways to incorporate this technology into their instruction. Several found ways to acquire Jasper or similarly useful videodisk software. In addition, each of the -FM' teachers was given a video camera so they could chronicle their assessment situations throughout the project. It also provided them with a "new tov." All but 1 of the 15 began to incorporate the camera into their daily class routines almost immediately. It provided them with a way for capturing classroom events, delivering information to students, parents, and colleagues, and documenting changes. By the end of the proje, t, 14 were consistently using the camera as an assessment tool. In several cases, involvement in 'FM' gave the teachers the credibility they needed t.) be part of the group at their schools who got the new toys. Their participation in the project demonstrated their capability for using the technology and sharing their expertise with others. For one of the middle school teachers, it meant going from a dal sroom with no computer to having a high-end Macintosh with an overhead panel and a notebook computer in one year. Prototypes. With the fast pace of technological innovation, new tools are always emerging. However, teachers rarely gain access to these cutting-edge technologies early enough to influence their adoption and/or use at their schools. Throughout the T'M' the teachers were given access to potential assessment tools such as Sunburst's Learner Profile. This accomplished several things: it encouraged it movation and exploration, it enabled the I'M' teachers to be involved in professional devolopment, and it provided informed feedback about the usefulness of the new tools to the educational community. For example, the Learner Profile gave the teachtrs a way to quantify their observations. While many hod previously lised a paper-based system for recording observations (a folder with individual "sticky notes" for each student), tlw bar code anu pen-based versions added several levels to their evaluations. With this tool, thev were able to easily generate reports by student or by skill area. This facilitated their planning as well as their progress reporting. However, the consensus %vas that the bar code reader was difficult to use. The alternative, interfacing with the Newton, added several hundred dollars to the cost, which put it beyond the affordability of most of the schools. The teachers' recommendation was that, while the concept of an electronic tool tor observation is useful, the technology itself still needs improvement. Methodologies The 'FM' teachers found that incorporating technology into their assessment repertoire dramatically changed their assessment practices. The technology altered the ways they delivered assessment information, their methods of data collec- tion, and their techniques for reporting information about their assessments. Delivery. Training during the TN' enabled many of the teachers to discover ways to use different technologies to provide sce- narios for assessment. They found diverse ways to present tasks, such as simulation exercises and audio taping tasks. Many learned how to use presentation software like Hyperstudio or PowerPoint to present mathematics tasks. Data Collection. Technology increased the teachers' capacity for accumulating indicators of studunt achievement. Somvtimes the ca nera served as a "third eye," allowing teachers to monitor one group of students while working with another gre.ap. All of tl.e teachers reported haying used the camera to sometimes gauge the thinking processes of their students. National Educational Computing Confi7ence, 1995 Pap' 50 5 The TIM' enabled 15 teachers to experience assessment with technology in a pragmatic, meaningful way. They were able to try out the technology, get assistance with the technology they already had, and share expertise with colleagues. The project team provided teachers with sustained opportunities to learn about the technology that enhanced their evaluation of students' achievements. The project also provided them with advocates and experts. This helped many of them make the case for their building administrators. technology with other teachers ai 'PM' empowered 15 teachers, providing them with easy access to technological tools, models of use, collaboration, and support. It helped diminish teachers' isolation and gave teir students access to new forms of assessment. It offered support for the creation of new, dynamic educational assessment. :t developed a strong network of active teachers willing to lead the way to new educational models. The group's continuation, through the dissemination of their work, will allow us to expand the network and provide teachers with leadership and continued support for alternative assessment. References Barrett, A. J. (1990, March). Objectively determining the educational potential of computer and video-based courseware; producing reliable evaluations despite the dog and pony show. Paper presented at the 7th International Conference on Technology and Education, Brussels, Belgium. Brophy, G. & Good, T (1986). Teacher behavior and student achievement. In M. C. Wittrock (Ed.), Handbook of research on teaching , (3rd ed., pp. 328-375). New York: Macmillan. Bunderson, C., Inouye, D. K., & Olsen, J. B. (1989). The four generations of computerized educational measurement. In R. L. Linn (Ed.), Educational Measurement, 3rd edition. New York: American Council on Education/ Macmillan. Carpenter, T. P. (1989). Teaching as problem solving. In R. I. Charles & E. A. Silver (Eds.), The teaching and assessing of math- ematical problem solving (pp.187-202). Reston, VA: The National Council of Teachers of Mathematics. Cobb, P. (1990). A constructivist perspective on information-processing theories of mathematical activities. International Journal of Educational Research, 14, 67-92. The Cognition & Technology Group at Vanderbilt. (1990). Anchored instruction and its relationship to situated cognition. Educational Researcher, 19(6), 2-10. Donovan, F., & Sneider, C. (1994). Setting and meeting the national standards with help from technology.Technology & Learning, 15(1), 40-44, 46, 48. Ginsberg, R., Sebastian, J., Underwood, J., Anderson, L., Kridel, C., & Stevenson, K. (1994, April). Teaching on television, teaching with technology: Discussing experiences in education courses. Division J Town Meeting at the annual meeting of the American Educational Research Association, New Orleans, LA. Greeno, J. G. (1989). For the study of mathematics epistemology. In R. I. Charles & E. A. Silver (Eds.), The teaching and assessing of mathematical problem solving (pp. 23-31). Reston, VA: The National Council of Teachers of Mathematics. Harel, I., & Papert, S. (1990). Software design as a learning environment, In I. Harel (Ed.), Constructivist learning (pp. 19-50). Cambridge, MA: MIT Press. Kaput, J. (1992). Technology and mathematics education. In D. A. Grouws (Ed.), Handbook of research on mathematics teaching and learning (pp. 515-556). New York: Macmillan. Natriello, G. (1987). The impact of evaluation processes on students. Educational Psychologist, 22, 155-175. Reynolds, A. (1992). What is competent beginning teaching? A review of the literature. Review of Educational Researth, 62,1-36. Scott, T.. Cole, M., & Engel, M. (1992). Computers and education: A cultural constructivist perspective. In G. Grant (Ed.), Revim of Research in Education (Vol. 18., pp.191-251). Washington, DC: American Educational Research Association. 1 This material is based upon work supported by the National Science Foundation under Grant No. ESI-9255790. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author and do not neces- sarily reflect the views of thi- National Science Foundation.