Di`CUMENT RESUME ED 362 185 IR 016 341 AUTHOR Marcinkiewicz, Henryk R.; Welliver, Paul W. TITLE Procedures for Assessing Teachers' Computer Use Based on Instructional Transformation. PUB DATE Jan 93 NOTE 7p.; In: Proceedings of Selected Research and Development Presentations at the Convention of the Association for Educational Communications and Technology Sponsored by the Research and Theory Division (15th, New Orleans, Louisiana, January 13-17, 1993); see IR 016 300. PUB TYPE Reports Evaluative/Feasibility (142) Speeches/Conference Papers (150) EDRS PRICE MF01/PC01 Plus Postage. DESCRIPTORS *Computer Assisted Instruction; *Educational Assessment; Educational Innovation; Educationaf Theories; Elementary Education; *Elementary School Teachers; Evaluation Methods; Models; Pilot Projects; *Teacher Attitudes; Teacher Role; *Technological Advancement; Test Reliability; Theory Practice Relationship; Use Studies IDENTIFIERS Paired Comparisons ABSTRACT The Levels of Computer Use assessment (LCU) was developed to classify teachers' use of computers so that variations of use could be studied. The LCU was derived from the model of Instructional Transformation of L. P. Rieber and P. W. Welliver. The model describes the process of the adoption of innovation, focusing on the adoption of the computer in education. It describes a teacher's five stages in the process of adopting the technology, from familiarization through the evolution that involves continuing practicing and leerning. The LCU was developed to assess computer use through the paired comparison technique. Pairs of comparisons were pilot tested with 50 elementary school teachers, and a revised version was field tested with 23 elementary school teachers and another sample of 170 elementary school teachers. The estimated reliability was high, and results indicate that the model of Instructional Transformation identifies at least two progressive levels, defined as integration and utilization. While the model is useful in understanding a teacher's theoretical progression in the adoption of computers, the assessment offers a means for applying the theoretical model. One table lists the instrument items and the levels they represent. (Contains 21 references.) (SLD) *********************************************************************** Reproductions supplied by EDRS are the best that can be made from the original document. * *********************************************************************** U.S. DEPARTMENT OF EDUCATION Deice of Educational Research and Improvement EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) C This document has been reproduced as received from the person or organization originating it 0 Minor changes have been made to improve reproduchon quality Points of view oi opinions stated fl this docu mere do not necessarily represent oft.cad 11) OE RI posawn Or pohcy (X) v( \ID C:1 cT1 Title: Procedures for Assessing Teachers' Computer Use Based on Instructional Transformation Authors: Henryk R Marcinkiewicz Paul W. Welliver "PERMISSION TO REPRODUCE THIS MATERIAL HAS BEEN GRANTED BY Michael Simonson l*".) or 679 TO THE EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC)." 3. Assessing Computer Use ...on Instructional Transformation The Levels of Computer Use assessment (LOU) was developed to classify teachers' use of computers so that variations of use could be studied. It was derived from the model of Instructional rfransformation (Rieber & Welliver, 1939; Welliver, 1990). The development of the LCU was stimulated by the identification of the discrepancy between advocacy for the use of computers in education and their actual use by teachers. It is widely believed that computer technology can help improve the educational system (see The National Task Force on Educational Technology, 1986; United States Office of Technology Assessment [OTA], 1988; Sheingold and Hadley, 1990; Shanker, 1990). At the same time, the availability of computers in schools has increased. In the period from 1983 to 1987, the average pupil-to-computer ratio in public elementary schools improved from 112.4 to 36.8 pupils per computer (OTA). Despite the positive trends in computer availability, relatively few teachers have integrated computers into teaching. A recent national survey has shown that even in schools where computers were available at more than double the national average, only about one teacher per school had integrated the computer into the classroom (Becker, 1989; Sheingold & Hadley, 1990). Apparently, sheer availability of computers did not result in their increased integration into teaching. Indeed, there is a strong sentiment of disappointment and disillusionment among teachers regarding educational technology (Becker, 1987; Bjerklie & Hollis, 19.c:1; Dronka, 1985; Ely & Plomp, 1988; O'Neill, 1990; Roblyer, 1988). The poor opinion of educational computing by teachers does not agree with the views of its supporters and patrons in government, academe, and teachers' unions. This discrepancy in perceptions of educational computing deserves study because the result of it is that computers in education are underutilized. As a result, whatever hopes there are for educational computing to affect education may never be realized. For the research on teachers' computer use, a fundamental question is, "What is meant by use?" This question has been addressed in the model of Instructional Transformation suggested by Rieber and Welliver (1989). Instructional Transformation The model of Instructional Transformation describes the process of the adoption of innovation. This is an area of human behavior that has been studied generally types of adoption behavior based on length of time have been described (Rogers, 1983; Rogers & Shoemaker, 1971) and specificallya framework for analyzing the adoption of innovations among educators has been described (Hall, Loucks, Rutherford, & Newlove, 1975). 'Me model of Instnictional Transformation focuses on the adoption of a specific educational innovationthe computerand describes a teacher's progression through five stages of involvement with computers (Rieber and Welliver, 1989; Welliver, 1990). First, a teacher becomes familiar with computers (familiarization); then, the teacher uses computers in teaching (utilization). A higher level of use is observed when the computers have become critical to the teaching (integration). An analogy to this scenario would be of one who upon learning to drive (familiarization), uses the car to improve one's general transportation in order to fulfill a variety of purposes (utilization). One may then schedule the use of the car so that the vehicle is critical to the function of fulfilling the purposes (integration). In education, this stage includes the teacher's awareness of the change of his or her role. With this new awareness of the restructuring of teaching activities, the teacher pursues an expansion and fine tuning of the computer-teacher-student relationship (reorientation). Tne final stage (evolution) is more of a suggestion than a condition, and that is to continue practicing and learning about how to develop the improvement of instruction through the systematic implementation of computer technology. The definition of use for the Levels of Computer Use assessment was derived from the model of Instructional Transformation. Use was defined as the integrated 1 Marciukiewicz & Welliver 6 80 Assessing Computer Use ...on Instructional Transformation employment of computers in teaching. Two levels of use were designatedUtilization and Integration. At the Utilization level, a teacher shares and delegates teaching duties to the computer; however, the absence of the computer would not prevent the implementation of instruction. At the Integration level, teaching duties would also be shared by and delegated to the computer; however, the absence of computers would prevent the implementation of instruction. The distinction between the Utilization and Integration levels lies in the expendability of the computer technology. Expendability describes the relationship of computer technology to a teacher's planned instructionwhether or not instrucdon would be able to continue in the hypothetical event of, say the sudden absence of computer technology. It is this diniension of expendability that was identified as the cut-off for membership in either category. According to the model of Instructional Transformation, the Integration stage is further characterized by the dimension of a teacher's emergent self- awareness of a role cl ange in teaching from teacher-centered to learner-centered. In a departure from the model, the assessment does not encompass the dimension of awareness of role change. Development of the Assessment The LCU is the result of several attempts at varying formats. In the original conception of the assessment, items were written for each of the five stages of the model of Instructional Transformation. This had been field tested and revised several times and was finally supplanted by the current format. The earlier forms of the assessment adhered to the progressive nature of the model and attempted to capture all of its dimensions, however, it was difficult to classify responses. This is because there are several dimensions that run throughout the model in a progressive overlapping manner. In the current form, one dimension is assessed. The most appropriate technique for this sort of assessment is the paired comparisons technique. This technique requires that a discrete definition be operationalized to determine the cut off for the levels of use. Additionally, the levels need to be mutually exclusive and exhaustive. Nunnally (1959) considers this method "the most exact psychophysical tool..." useful for precise information concerning judgments or preference. In the format of the LCU, each item from one level is paired alternately with each item from the other level (See Table 1). For example, if there were four items, two from the first level (la, lb) and two from the second level (2a, 2b), the paired comparisons would be (1a/2a), (1a/2b), (1b/2a), and (1b/2b). The respondent is forced to endorse one item per pair. The paired comparisons technique allows only one combination of responses to indicate membership at either level. There are four pairs of items in this version. Two items indicate the Utilization level and two the Integration level. If, for example, a value of 1 were assigned to responses indicating the Utilization level, consistent responses for 4 items would score 4 points. A value of 2 for responses indicating the Integration level, would result in consistent response scoring of 8. Acceptable scores would be 4 or 8. Scores of 6 or 7 indicate an inconsistency. The four items were selected from a field testing with a sample of 50 " elementary school teachers because those items had the highest response consistency. In that testing, there were 15 items. Items which generated response inconsistency or which introduced an additional dimension were eliminated. In a subsequent field testing, twenty-three elementary school teachers responded; there were 2 inconsistent responses. The estimated reliability of the LCU for the second field trial using the Coefficient of Reproducibility ( CR) was .74. One of the items was reworded. The LCU was then administered to 170 elementary school teachers. The CR was estimated at .96. A CR of .90 is the criterion for demonstrating that items form an ordered scale of allowable response patterns (Crocker & Algina, 1986). The teachers were also asked to respond to a control item about their self- reported computer use; the responses were matched with those of the LCU and 2 Marcinkiewicz & Welliver 681 Assessing Computer Use .on Instructional Transformation provided additional data for estimating criterion-related validity. The consistency of classification of the measure was estimated computing Cohen's kappa (Kappa = .72). The results are important for two reasons: 1) the estimated reliability is high, and 2) the demonstration of an ordered scale suggests that the model of Instructional Transformation identifies at least two progressive levels. Table 1 The Levels of Computer Use assessment In my instruction, the use of the microcomputer is supplemental. *a. 1. b. The microcomputer is critical to the functioning of my instruction. 'The use of the microcomputer is not essential in my instruction. a. 2. b. 1 or my teaching, the use of the microcomputer is indispensable. a. The microcomputer is critical to the functioning of my instruction. 3. The use of the microcomputer is not essential in my instruction. *b. a. For my teaching, the use of the microcomputer is indispensable. 4. In my instruction, the use of the microcomputer is supplemental. *b. indicates an item of the integration level. Note: * indicates an item of the utilization level. The LCU has been used in several studies. It has been used to establish a dependent variablelevel of computer usein a study of elementary school teachers (Marcinkiewicz, 1991, in press) and hi a longitudinal study of preservice undergraduate teachers (Marcinkiewicz & Grabowski, 1991). It has been adapted by the Grosse Pointe (Michigan) Public School System for use in its Staff Computer Skills Survey (J. Corbett, personal communication, February 21, 1992). It is being used to establish a dependent variablelevels of computer use --relative to the subjective norms of rural teachers. It should be noted that prior to this writing, the assessment has been referred to as the LU assessment. As of this discussion, the assessment is referred to as the LCU Levels of Computer Use --assessment. This modification has been made to reflect the purpose of the assessment more accurately. This modification also underscores the ongoing study of the assessment and of the model of Instructional Transformation. Discussion There is a fundamental notion in computer implementation that is often overlookednot all intended computer users do use them. As with most innovations, people vary in their adoption of computer use. By focusing on a specific profession such as teaching, the study of how its membersteachersdiffer in their use of computers is more manageable. They differ before they begin to use computers and they differ once they have begun to use them. Before use, they differ in the time it takes to adopt the idea of using the technology. After teachers adopt the notion of using computers, they differ in the nature of their computer use. The importance of the model of Instructional Transformation is that it addresses the fundamental variance in the adoption of computer use. It describes the progression of a teacher from the onset of the adoption of computers. In application, this theoretical model will help educators hi several ways: 1) to identify the range of expected behaviors 3 Marcinkiewicz & Welliver 682 Assessing Computer Use ...on Instructional Transformation applicable t..) a teacher's computer usewhat can we expect a teacher to become?; 2) to identify the level of a teacher's computer use; 3) to identify whether a teacher is progressing in computer use; 4) to identify how the nature of a teacher's instruction changes due to his or her computer use; 5) to identify how the nature of teacher's professional self-perception changes due to computer use. When educators can identify these aspects of teachers, the professional growth and development of teachers relative to computer use can be understood. With this understanding there is a basis for recommending staff development, remediation, or differential staffing. The LCU assessment was developed so that educators could implement the model of Instructional Transformation. Some applications of the assessment include assessing a school's teachers to determine the current computer utilizationlunderutilization levels before a purchase order for computers is issued, so that the likelihood of future use can be estimated. Also, as a school is designing its restructuring plans, the LCU can be used in needs assessment to identify discrepancies in the levels of computer use. A school may also study the assessment and the model of Instructional Transformation to learn what the extent of possibilities of computer use in education are. In doing so, a school can be helped in clarifying its goals in the application of computer technology. The LCU accomplishes the classification of teachers into two levels of computer use. When used for this purpose, the assessment addresses point #2 mentioned above. The current form of the LCU is markedly different from earlier versions. The way in which the current and past forms differ needs to be pointed out to highlight the strength of the current version as well as to identify directions for future research. Most importantly, the past and current versions differ in that the former comprised several dimensions as does the modelattempting to measure all dimensions. This proved to be confounding since the dimensions share characteristics. It was decided that it was not practical or perhaps possible to capture all the dimensions simultaneously. In the LCU assessment, there is one dimension the expendability of computersthat is measured. This is the central dimension of the model. Measures of the other dimensions need to be developed; such as for the degree of awareness of the changing of one's role a a teacher. The successful development of the LCU establishes it as a part of the assessment arm of the model of Instructional Transformation. The model is useful in understanding a teacher's theoretical progression in the adoption of computers. The assessment offers a means for applying the theoretical model. References Becker, H. J. (1987). The impact of computer use in children's learning: What research has shown and what it has not. (Report No. 18) Baltimore, MD: The Johns Hopkins University, Center for Research on Elementary and Middle Schools. Becker, H. J. (' 989). [U. S. Administration of the 1989 IEA computers in education survey, Johns Hopkins University Center for Social Organization of Schools]. Unpublished raw data. Bjerklie, D. & Hollis, R. W. (1991, May 20). The revolution that fizzled. Time . pp. 48-49. Crocker, L. & Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart, and Winston. Dronka, P. (1985). Computer integration into instruction is stuck: Experts blame unclear optimal uses and three implementation problems. ASCD Update. 1-8. Ely, D. P., & Plomp, T. (1988). The promises of educational technology. In B. Branyan-Broadbent & R. K. Wood (Eds.), Educational media and technology yearbook 1988, 5-18. Englewood, CO: Libraries Unlimited. 4 Marcinkiewicz & Welliver 683 Assessing Computer Use ...on Instructional Transformation Hall, G. E., Loucks, S. F., Rutherford, W. L., & New love, B. W. (1975). Levels of use of the innovation: A framework for analyzing innovation adoption. The Journal of Teacher Education, 26(1), 52-56. Marcinkiewicz, H. R. (in press). Computers and teachers: Factors influencing computer use in the classroom. Journal of Research on Computing in Education. Marcinkiewicz, H. R. (1991). The relationships of selected personological vaqables to the use of available microcomputers by elementary school teachers. Doctoral dissertation, The Pennsylvania State University. Marcinkiewicz, H. R. & Grabowski, B. L. (1991). The relationships of personological variables to computer use by elementaty school teachers: Report of Phase OneBaseline Data. Proceedings of Selected Research Paper Presentations 1992, AECT. National Thsk Force on Educational Technology. (1986). Transforming American education: Reducing the risk to the nation, a report to the Secretary of Education, United States Departmert of Education. T.H.E. Journal, 14(1), 58-67. Nunnally, J. C. (1959). Tests and measurements: Assessment and prediction. New York: McGraw-Hill. O'Neill, J. (1990). Computer 'Revolution' on hold: Despite scattered progress, few signs of broad-based changes seen. ASCD Update. 32(9), 1-5. Rieber, L. P. & Welliver, P. W. (1989) Infusing educational technology into mainstream educational computing. International Journal of Instructional Media, 16(1), 21-32. Roblyer, M. D. (1988). The effectiveness of microcomputers in education: A review of the research for 1980-1987. T. H. E. Journal, 16(2), 85-89. Rogers, E. M. (1983). Diffusion of innovations. (3rd ed.). New York: The Free Press. Rogers, E. M. & Shoemaker, F. F. (1971). Communication of innovations: A cross- cultural approach. New York: The Free Press. Shanker, A. (1990, September 23). Equipping teachers for productivity: From (Section E), p. 7. chalkboards to computers. The New York Times , Sheingold, K. & Hadley, M. (1990). Accomplished teachers: integrating computers into classroom_practice. New York: Bank Street College, Center for Technology in Education. U. S. Congress, Office of Technology Assessment, Power on! New tools for teaching and learning, OTA-SET-379, Washington, DC: U.S. Government Printing Office, September, 1988. Welliver, P. (1990). Instructional transformation: A model for change. (The Pennsylvania State University Regional Computer Resource Center report to the state centers and member teachers). University Park, PA: ITEC Center at Penn State . 1'1 5 Mareinkiewicz & Welliver 684