Navy Personnel Research and Development Center San Diego, California 92152-6800 TR-91-4 January 1991 / Low Cost Microcomputer Training Systems Project, Computer Based Educational Software System: Final Report C. Douglas Wetzel W.H .W ulfeck II F E Af e pl1 2 1991 Approved for public release: distribution is unlimited. 91 2 1I1 NPRDC-TR-91-4 January 1991 LOW COST MICROCOMPUTER TRAINING SYSTEMS PROJECT, COMPUTER BASED EDUCATIONAL SOFTWARE SYSTEM: FINAL REPORT C. Douglas Wetzel W. H. Wulfeck 11 Reviewed by Janet L. Dickieson Approved and Released by B. E. Bacon Captain, U.S. Navy Commanding Officer and R. C. Sorenson Technical Director (Acting) Approved for public release; distribution is unlimited. Navy Personnel Research and Development Center San Diego, California 92152-68(X) REPORT DOCUMENTATION PAGE Form Approved I OMB No. 0704-0188 Public reporting burden for this collection of mformauon is estunated to average I hour per response including the time for reviewing instructions, searching existing data sources, gathcnng and maintaining the data needed., and completing and reviewing the collecton of mformation. Send comments regarding this burden estmate or any other aspect of Lhisc ollection of informauion, including suggestions for reducing this burden, to Washington Headquarters Services, Directirate for Information Operations and Reports, 1215 Jefferson Davis Ifghway. Suite 1204. Arling ton. VA 22202-4302. and to the Office of Managemnent and Budget, Paperwork Reduction Project (0704-01 88). Washington. IX. 20503. 1 AGENCY USE ONLY (Leave blank) 2. REPORT DATE 3. REPORT TYPE AND DATE COVERED I January 1991 Final FY82-FY90 4 TITLE AND SUBTITLE 5. FUNDING NUMBERS Low Cost Microcomputer Training Systems Project, Program Element 0603720N Computer Based Educational Software System: Final Report Work Unit Z-1772-ET002 6. AUTHOR(S) C. Douglas Wetzel and W. H. Wulfeck I 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION Navy Personnel Research and Development Center REPORT NUMBER San Diego, California 92152-6800 NPRDC-TR-91-4 9 SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSORINGMONITORING Chief of Naval Operations (OP-11), Navy Department, AGENCY REPORT NUMBER Washington, DC 20350-2000 11 SUPPLEMENTARY NOTES 12a DISTRIBUTION/AVAILABILITY STATEMENT 12b. DISTRIBUTION CODE Approved for public release; distribution is unlimited. 13 ABSTRACT (Maximum 200 words) The Low Cost Microcomputer Based Training Systems project addressed the amount of development effort required to create computer-based instruction (CBI) by instructional developers, the proliferation of nontransportable machine-specific CBI software over incompatible hardware systems, and the opportunity to standardize previously successful CBI software using generative techniques. The project conducted assessments of Navy training patterns relative to CBI, developed the Computer Based Educational Software System (CBESS), and fielded CBESS at representative test sites. CBESS standardizes a set of CBI development and delivery tools in five subsystems which include general CBI programs, a management interface, and three specialized subsystems involving fact memorization, technical vocabulary, and equipment simulation. Instruction developed at test sites spanned several content areas, including threat fact memorization with a videodisc, study skills, mathematics, and electricity and was employed as a supplement to instructor resources for remediation, refresher training, self-study, and repetitive practice 14. SUBJECT TERMS 15 NUMBER OF PAGES Computer based instruction, authoring, computer aided instruction, computer based training 88 IC PRICE CODE 17 SECURITY CLASSIFICA- 18 SECURITY CLASSIFICA 19. SECURITY CLASSIFICA- 20 LIMITATION OF ABSTRACT TION OF REPORT TION OF THIS PAGE TION OF ABSTRACT UNCLASSIFIED UNCLASSIFIED UNCLASSIFIED UNLIMITED NSN 7540-01-280-5500 Standard Form 298 (Rev. 2-89) Prescribed by ANSI Std. Z39- IS 298-102 FOREWORD This work was funded as part of the Advanced Development project entitled Low Cost Micro-computer Training Systems (Program Element Number 0603720N, Work Unit Number Z-1772-ET002). The project was the result of an operational requirement originally promulgated by the Chief of Naval Operations (OP-987H, OP-01B7) and then subsequently supervised by OP-11. The purpose of this research was to assess Navy requirements for computer-based i.struction id de--!op ccipu .- niuuion application software tor wide Navy application through tryouts at representative test sites. The results of the project are primarily intended for the Department of the Navy training community. B.E. BACON RICHARD C. SORENSON Captain, U.S. Navy Technical Director (Acting) Commanding Officer Acoession For NTIS GRA& I DTIC TAB Unannounced [ IJustification C By_/ Distribution/ Availability Codes Avail and/or Dist Special -v- SUMMARY Problem This project was concerned with Navy training needs for authoring and delivering instruction on low-cost microcomputer-based training systems. The problems addressed were the amount of development effort required to create computer-based instruction (CBI) by instructional developers, the proliferation of nontransportable machine specific CBI software over incompatible hardware systems, and the opportunity to further develop and standardize promising ideas from previous research and development work in sophisticated generative approaches to CBI. When the project began in 1982, CBI often took the form of specialized computer programs with instructional content embed- ded in the programs themselves. CBI generally required extensive develcpment time and high levels of computer programming expertise that exceeded that of most instructional developers. Incompatible hardware and operating systems created serious transportabil- ity problems such that instruction developed for one machine often would not run on oth- ers without expensive recoding. At the outset of the project, the risks involved turning ideas from previously developed unstandardized programs into easily usable technology, and risks associated with hardware and software engineering market forces leading to future standardization. Additionally, successive cost reductions in what is still an emerg- ing field offered the technological opportunity to provide the Navy with automated instruction, remediation, and drill and practice to supplement instructor resources. Purpose The overall purpose of the project was to provide the Navy with automated tools for developing computer-based instruction. Th,; project sought to standardize a set of computer-based instruction strategies into a system and to reduce the expertise required to produce instruction. This report describes the status of the project after these tools were developed and fielded at representative test sites. Approach The overall approach of the project involved three elements: (1) assessment of Navy instructional practices relevant to CBI, consisting of a survey of instructional managers and a tabulation of the frequency of various instructional objectives, (2) development of CBI software for wide Navy application which resulted in the Computer Based Educational Software System (CBESS), and (3) development of demonstration test-beds for the CBESS authoring and delivery systems that actualized Navy specific courseware at various Navy training sites. Results and Conclusions One standard system was created from a diverse set of software which had previ- ously been prototyped in various programming languages for different hardware plat- forms using divergent standards. The resultant difficulty level of the system decreased - vii - relative to tne prototypes from which it was derived. This work resultcd in formal authoring tools which moved the authoring of computer based instruction from the realm of progranumers into that of instructional developers. The CBESS developed consists of five subsystems: (1) the Computer Based Memorization System (CBMS), which is specialized for repetitive fact training and has been used with large threat databases and a videodisc; (2) the Equipment Problem Solv- ing Trainer (EPST), which is specialized for equipment simulation in the context of locating and replacing faulty parts; (3) the Language Skills Computer Aided Instruction (LSCAI), which is specialized for technical vocabulary training; (4) a General Computer Based Instruction (GCBI) package, which is a flexible general purpose utility for creating unique interfaces and lesson sequences; and (5) instructional management program., which provide a menu interface linking the lessons from the other four packages. CBMS, EPST, and LSCAI are specialized authoring facilities that reduce development tirite by assuming certain pre-configured instructional delivery strategies. Authoring and student programs were designed to separate courseware from execut- able programs so that the system can be reused to create many new varieties of separate instructional courseware lesson files. The authoring programs use standardized self- contained editors that reduce the effort of instructional developers and now make the availability and capability to produce CBI more widespread. The skill required by the programs generally assumes some prior basic operating knowledge of computers and instructional design. Market forces during the project reduced the number of prominent standard computers, which reduced the need to recode CBI among hardware platforms. The programs were specifically adapted to Navy standard microcomputers and can be reconfigured over a range of hardware options, such as display cards and videodisc players. The system was formally documented in 18 user manuals and the government controls the source code and can updaite it with desired features in the future. The CBESS was successfully used by developers in creating deployable instruction that now remains at various Navy training commands as a regularly used instructional media. Four of the CBESS packages were used in substantial development efforts and a catalog of instruction documents these finished products. The incremental development of the system was responsive to user needs through an ongoing program of modifications, updates, and user training. System modifications during field tests resulted in an increase in the ease of using the programs and an increased utility with newly added features. Software development records showed 43% of the modifications traceable to user suggestions, with 70% of those being related to interests in interface features. The CBESS is applicable across many ratings and for many types of instructional content. The developed lessons and potential applications include: remediation, enhancement, refresher, reviews, initial primary instruction, repetitive drill and practice, self-study, and as a general supplement to instructor resources. The developed lessons generally addressed specific training objectives supplementing larger bodies of regular course material. The system was successful in the intended application environments as indicated by its regular use by students and instructional managers, its contribution to increased performance or reduced attrition, and by the desire of test-bed sites that it be continued or expanded, and supported in the future. - viii - Several evaluations are reported. Surveys showed heavy emphasis in Navy training for fact and procedure type learning objectives, and course managers reported concerns with curriculum stability and student entering skills. Student performance results from several test sites and previous studies included higher progress test scores, fewer retests, less training time, reduced attrition, fewer set backs, and increased usage with material tailored to course quizzes and supplemented by a videodisc. One intensive study of the authoring process with the specialized LSCAI showed reasonable development times and actualized a decision matrix method for selecting courses that would most benefit from computerization. Recommendations The following recommendations are for OP-I1 and the Navy education and training community: 1. Continuing life cycle management support should be given implementation atten- tion to realize previous development investments in the government-controlled CBESS. The success of the project directly implies specific post-project maintenance to support the continued operational use of the system and developed instruction. 2. Support responsibilities should be assigned and funding should be sought from a broad base appropriate to the wide number of applicable ratings. 3. CBESS can be adopted as a standard to avoid proliferation of incompatible and nontransportable lessons. Exceptions to the use of government-controlled CB1 software should be allowed for justified special capabilities. The implementation of CBESS should proceed at sites such as the CNET Model Schools program. 4. User support should be rrovided for distributing software, manuals, maintaining stock, and consulting that is tailored to the intended instructional development purpose of the software. 5. Routine software life cycle maintenance should be planned to continue the viabil- ity of the CBESS on new host devices and to preserve investments in previously developed CBI so that it can continue to be delivered and updated. 6. Existing instruction in the CBMS threat databases should not be allowed to go out of date and should be maintained and updated centrally because of its wide applica- bility. 7. The Navy should systematically guide computer-based instruction technology by fostering the support infrastructure required by an environment with inherent personnel rotations and loss of trained individuals. Instructors will remain consumers and need resource specialists similar to those that have evolved in civilian school systems or currently exist in audio-visual support specialists. 8. Computer-based instructional development efforts should employ decision ci- teria that consider student throughput, course stability, course importance, level and type of training cbjectives employed, potential of remediation to affect attrition or setbacks, management potentials such as supplementing instructor resources, and basing the selec- tion of appropriate software tools on instructional requirements. - ix - 9. Computer-based instruction technology continues to evolve and the Navy should adapt the CBESS to new DoD portability standards and enhance these systems with further reductions in user skill requirements. Although authoring systems distance developers from low level programming, developing computer-based instruction still requires more expertise than does developing conventional instruction. "X - CONTENTS IN T R O D U CT IO N ....................................................................................................... . . Problem and B ackground ...................................................................................... I P u rp o se ............................................................................................................ . ..... 2 A PP R O A C H ................................................................................................................ A ssessm ent Surveys ............................................................................................... 2 Development of Computer-based Instruction Software ...................................... 4 Previous Wo rk ................................................................................................ 6 Standardization C ontract ................................................................................ . 6 Post-contract In-house Work .......................................... 8 Description of CBESS Programs ......................................................................... 8 Computer Based Memorization System (CBMS) ........................................... 8 Equipment Problem Solving Trainer (EPST) ................................................. 11 Language Skills Computer Aided Instruction (LSCAI) ................................. 12 G eneral C Bi Package (G C BI) ......................................................................... 13 Instructional Management Programs .............................................................. 15 Target Hardware and User Manuals .............................................................. 15 G enerative Features of C BESS ........................................................................ 15 D m- srr at'- T est-bed , . . ........... . . . ...................................................... 19 TEST SITES AND INSTRUCTIONAL DEVELOPMENT ...................................... 20 E V A L UA T IO N S ...................................................................................................... . 34 Analysis of Software Development Effort ............................................................ 34 Study of CBI Decision Criteria and LSCAI Authoring ....................... . 37 LSCAI in the Academic Remedial Training Program ......................................... 38 Effectiveness of LSCAI in the Memphis JOBS Program ..................................... 38 Operations Specialist (OS) A-school LSCAI Data ................................................ 40 Model EM A-school NTC Great Lakes ................................................................ 43 T hreat M em orization Training ............................................................................. 44 CBMS Program Preference and Database Usage ........................................... 45 Repeated U sage by Individuals ....................................................................... 47 Learning Patterns in One Domain ................................................................... 47 Tailoring Databases to TAO Course Content ................................................. 50 T A O C lass S tanding ...................................................................................... 5 1 C B M S in G ro up s ............................................................................................. 5 1 Equivalence of Computer and Paper-based Threat Memorization Tests ........ 54 -xi - DISCUSSION ........................................................ 55 Accomplishments During the Project..................................................... 55 Observed Patterns and Lessons Learned ................................................. 56 When is CBI Appropriate to Use ..................................................... 56 Types of Instruction Developed....................................................... 57 Quality of Instruction Developed ..................................................... 59 Role of User Experience in Creating CBI ........................................... 60 Problems and Components of Successful Endeavors.............................. 61 CBI is Still an Emerging Technology ................................................ 62 Infrastructure for Computer-based Instruction ...................................... 64 Implementation and Life Cycle Maintenance ........................................... 64 Software Maintenance ............................................................. 65 User support............................................................................. 65 Courseware Maintenance and Threat Database Updates .......................... 65 New Instructional Development ...................................................... 66 Post-project Maintenance Mechanisms .............................................. 66 RECOMMENDATIONS....................................................................... 68 REFERENCES ................................................................................ 69 APPENDIX A--CBESS Manuals................................................... .......... A-0 DISTRIBUTION LIST -XII -