DOCUMENT RESUME TM 820 683 ED 222 532 Ree, Malcolm James; And Others AUTHOR Calibration of Armed Services Vocational Aptitude TITLE 9, and 10. Battery Forms 8, Air ForceiHuman Resources Lab., Brooks AFB, Texas. INSTITUTION AFiTL-.TR81-49 REPORT NO PUB DATE Feb -S-2. NOTE 19p. MF01/PC01 Plus Postage. EDRS PRICE *Aptitude Tests; *Armed Forces; Enlisted Personnel; DESCi'IPTORS *Equated Scores; Test Reliability *Armed Forces Qualification Test; *Armed Services IDENTIFIERS Vocational Aptitude Battery; Calibration ABSTRACT A calibration of the Armed Forces Qualification Test (AFQT) composite of the Armed Services Vocational Aptitude Battery (ASVAB) Forms 8a, 8b, 9a, 9b, 10a, and 10b to the metric of the AFQT Form 7a (AFQT-7a) and a comparison of these outcomes to the operational calibration tables implemented 1 October 1980 are presented. A sample of applicants for military enlistment was administered one form of ASVAB and the AFQT-7a in counterbalanced order. For analytic purposes, an edited sample (15,115 males) was separated into 6 samples based on the 6 forms of ASVAB administered. Data were collected at 20 geographicallSr dispersed Armed Forces Examining and Entrance Stations on the 6 forms of ASVAB and the AFQT-7a. Each of the six samples was edited and scored, and descriptive statistics were computed. The root-mean-square and investigating the similarity of average absolute deviation measures, differences the equated scores across the forms, showed only small during this study. among the operational table and tables developed 9, and 10 of ASVAB were found to be parallel when equated to Forms 8, AFQT-7a, and a single conversion table was deemed appropriate for operational enlistment processing. (Author/PN) *********************************************************************** Reproductions supplied by EDRS are the best that can be made from the original document. *********************************************************************** U.S. DEPARTMENT OF EDUCATION PERMISSION TO REPRODUCE THIS NATIONAL INSTITUTE OF EDUCATION MATERIAL HAS BEEN GRANTED BY EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) This do,ument lids been reproduced as received from the person Or organization AFHRL-TR-81-49 orioin3hnq it PL2.4-e,Wrc.) thlen made to improve %nor chanqes tidy. reproduction quality TO THE EDUCATIONAL RESOURCES Pnts of view or opinrons stated In this docu INFORMATION CENTER (ERIC)" rhent do not neces,drilv represent official NIE pUltt y por;ilwr %IR FORCE CALIBRATION OF ARMED SERVICES VOCATIONAL APTITUDE BATTERY FORMS 8, 9, AND 10 U. By Malcolm James Ree John J. Mathews Cecil J. Mullins A Randy H. Massey, Capt, USAF MANPOWER AND PERSONNEL DIVISION Brooks Air Force Base, Texas 78235 February 1982 July 1981 Interim Report for Period October 1980 0 public release: distribution unlimited. Approved for LABORATORY COMMAND AIR FORCE SYSTEMS 78235 BROOKS AIR FORCE BASE,TEXAS 2 NOTICE When Government drawings, specifications, or other data are used for any purpose other than in connection with a definitely Government-related procurement, the United States Government incurs that the Government may have formulated or no responsibility or any obligation whatsoever. The fact in any way supplied the said drawings, specifications, or other data, is not to be regarded by implication, or otherwise in any manner construed, as licensing the holder, or any other person or corporation; or as conveying any rights or permission to manufacture, use, or sell any patented invention that may in any way be related thereto. is releasable to the National Technical The Public Affairs Office has reviewed this report, and it Information Service, where it will be available to the general public, including foreign nationals. This report has been reviewed and is approved for publication. NANCY GUINN, Technical Director Manpower and Personnel Division RONALD W. TERRY. Colonel, USAF Corn ma nder 3 m SECURITY CLASSIFICATION oF THIS PA'GE (When Data It:Wet-ea', READ INsTRucTioNs REPORT DOCUMENTATION PAGE BEFOP.E CONIPLETING FORM RECIPIENT's cATALOG NUMBER REPORT NUMBER 12. GOV'T A-CES5ION NO 3 1 \ I IMI - r11-81. Pi , TYPE OF REPORT ti PERIO:) (-ovERE.O TITLE land Subirtle, 5 4 Ill106111 \ " \lift\ ki [Ilk \ Ti( )\ 1 11 \ 1011.1) ,FIl \ l(:I: \ I (leinluir 1980 - Juls, 1981 I \ NI) III \ l' 1.1.11 Id. II \ III- lik II)11 \IS 8. 9. PERFORMING 3-20, REPORT NUMBER 6 CONTRA:7T 7)R GRANT NUMBER, s, k.:TmoR, , 8 7 \folio!, uieil .1. uI111 .Idnio, HIN, \latheli. Hands, .11(111, .1 FROGRAM ELEMENT, PROJECT. T pERcoRmIN-.; oRGANIZATION NAME AND ADDREss 13 .3 AREA a WCRK utUT NUMBERS 111(1 1)(4...I'M" III""1"" uritFI 62.71131' I .1'4, IIIIIII,III liemlu rrr, dit IN km I :7191801 I uric !him-. I;tnuls. I.,^ \J.- 7823.; \ II 12. REPORT DATE coNTROLLING :)FFI.CE NAME AND ADDRESS I February 1982 \ ht.) 1 ir 1.-nre, Human 81--nurri-- kallorilINF\ I 11 ( NUMBER OF PAGES kir 1 3 1 c.d.. 7821., 1 loll. lid...e. Brook 18 this rep,ri c,ntroilrn (of SECURITY CLASS. I5 1(1,-e) MON n- GRIN -.3 AGEN ---Y N AME a ADDRESSof different fro 4 I (lila-11)rd Is., DECLASSIFICATION DOWNGRADING SCHEDULE it rills Report) DISTRIBHTION sTATEMEN 6 111-trilonlon unliontrd. \ pprio,eil tor !whin wird, DISTRIBUTION STATEMENT 'of the abstract entered in Block 20, it different horn Report, 7 SUPPLEMENTARY NOTES IS ,\I 11d\ Numhtir 7728 KEY WORDS (Continue on reverse side if n cessary and Went!! y by block number) 19 Ir,IIIIV clIrs, Y. ,III0BIIIIIIg i1111105 ii(platillg ' \ 1:1,Y1 rquipercimillIr dploiolii to..d.- \s\ \II IIIII 110111I,II rPgre"IIIII ,dIihralion ABSTRACT 'Cont(nue on reverse side If necessary and identify by block number) 20 \ I'VF) compoSilli 4 Ow \ ruled ,-jers, ice, 'no' ohpieliSti IIii, In ealihrdh, [Ile \ Filled 1011.1, QIIiIhriCililull Te.t I ( \S\ \ In Form. 8a. 8b. Oa, Oh. Ma. and IDli lo Ole imilrie nf I lie \ 0,ational \ plitodo ILIIIvrs, \ 1:QT Form Ta \ ,ailiple of ()ember I980. 7,0 dm! In cullipan, !hemn uilIculne, lo the nperatinual valihrdlinn table, implementrd 1 dpplicont . 1nr iiiiIitdo hildisImmit sAa.- admini,11ire1i nor form of k-; \ \ It and 1111, kryT-Ta ill vntiotcrhalaileed order. wide.- (Ink- 511(19)1r ul 1.1 15 11i1- (II'S ehyell IIhrmigh data eidillog techniques I row this 1drgel ...,ouphi Id 22.11M, a ile..iglipil to ovitide Itimah-- and l'a,V- SA lill 1111'0111111111' PI' IIIIIPabie data. For ;mak tit purpo,cs. Illi, edited ...ample sAa, fp,Irilit'd \ 8 adillini,lered. Data mere collected at 20 wingrapliit all \ 1,,,,I BB Olt' "IS fOriII, of \ 11110 "IS si111111;1`, 1 \ S \ .\ 11 and Ille \ ruled Imre,'" E \ atilining and EIIIranrh. SISIION. ( \ 1:1-4;;) (ill Ow ..iS form. of \ 1.1,1T-Ta. 111,1er-lid EDITION OF I NOV 65 IS OBSOLETE 1473 DO FOR" 1 JAN 73 1 SECURITY CLASSIFICATION OF THIS PAGE (When Dta Enter.d) t nelassified SECURITY CLASSIFICATION OF THIS PAGE(Whon Dat Entorad lontimi(d): Item 20 males onl) .amples Mill% edited and scored. and descriptive statistics \very computed. Vereentiles for Kach of the smoothed b) a pol.)notilial regression procedure, F..ach sample both the 1S1 113 and the 11V1-1a mere equated mas plit in half. and the equating and smoothing mere repeated on each half sample. Since results mere consistent among the large sample and the tmo half samples. the) were accepted. In order to in estigate the similarit) Of the equated scores aeross the forms. root-mean-square (liNtS) and average absolute d('viation (1A1)) measures were the) 1 comparison of the forms found them to be equivalent N1 computed betmeen the various equating table,. mere equate(i to 1i,QT-7a. Th, 1iN1S and 1 11) measures showed onl) small differences tinning the operational table and table, de\ eloped during this stud). Forms R Q. and 10 of 1S) 113 Nvere found to be parallel when equated to rcry-Ta. and a .ingle conversion table Mil:, deemed appropriate for operational enlistment processing. I. nclassified SECURITY CL ASSIFICATION OF T..` P AGE(Whon Data Entorad) PREFACE This study was completed under the auspices of Personnel Qualification Systems which is part of a larger effort in Force Acquisition and Distribution. It was substinwd under project 77191801. Maintenance and Improvement of Enlisted Selection and Classification Tests.' and executed as part of the responsilnlity of the Air Force Human Resources Laboratory (AEHRL) as lead laboratory under the executive agent (Air Force) for Armed Services Vocational Aptitude Battery research and development. An effort such as this. ahhough under the direction of an individual. can IM` accomplished only through Jenny a team effort. The authors wish to express their appreciation to Roy Cho Ilman. James Ear les. Al C Hodge, and A 1C Gerald Yates. A debt of gratitude is owed to Doris Black, who served to condense and translate the analytic requests into operational procedure for the Technical Services Division. Henry Clark wrought minor magic by convituing the computer to produce analyses prior to established due dates. The authors also wish to express their appreciation to Jacohina Skinner and the other nwmbers of Pubhcation Review Panel 2 for helpful comments on an earlier draft of this manuscript. TABLE OF CONTENTS Page Introduction 5 I. Calibration of Tests 5 6 Method II. 6 The Tests Administration of Tests to Subjects 7 8 Data Editing 8 Sample Equipercentile Equating and Calibrating 8 9 Table Generation Results and Discussion 10 III. 10 Data Editing 11 Analysis Samples 12 Descriptive Statistics 12 Equating Tables for the AFQT Forms 12 16 Conclusions IV. 16 References LIST OF ILLUSTRATIONS Page Figure Scatter Plot of Arithmetic Reasoning and Numerical Operations Test Scores 11 I LIST OF TABLES Page Table 6 Name and Number of Items for Power and Speeded ASVAB Subtests in Forms 8, 9. and 10 i AFEES Sites and Target Sample at Sites 7 2 9 Example of Smoothing by Polynomial 3 Number of Subjects Flagged by Key Verification by Test Form 10 .1 10 Example Cases from Key Verification Procedures 5 12 6 Number of Subjects by ASVAB Form Descriptive Statistics for ASVAB 8, 9, and 10 and AFQT-7a 12 7 Conversion Tables for Each Form 13 8 9 Deviation Measures Comparing Use of One Versus Six Conversion Tables 15 Classification by Mental Category Based on One Versus Six Tables 15 I() 16 Deviation of Percentile Scores across Category Lines 11 7 3 CALIBRATION OF ARMED SERVICES VOCATIONAL APTITUDE BATTERY FORMS 8, 9, AND 10 I. INTRODUCTION The measurement of human characteristics has been a necessary part of selection and classification for mili- tary occupations for over 60 years. Like measurement of physical characteristics, such as length, weight, or density, no natural units of measure exist for psychological characteristics; rather, artificial units are established by consen- sus. One of the most frequently used units of measurement for human characteristics is the percentile equivalent. The percentile is reported in reference to some standard population or group. Ability tests used for military selec- tion and classification are usually referenced to the 1944 mobilization base, and this is usually accomplished by equating new tests to old tests. Equating is the conversion of score units of one test tu the score units of another test. The current study describes the referencing of Forms 8a, 8b, 9a, 9b, 10a, and 10b of the Armed Services Voca- tional Aptitude Battery (ASVAB) to the mobilization base metric, through the use of an anchor test. There are two important reasons why current tests are equated to past tests. The first is to enable the testing agency to report on the relat;ve distribution of scores on a year-to-year basis in a common metric. For example, the various military services lice to be able to compare current accessions to past accessions on the same scale. The second reason is to provide a consistent meaning for cutting scores for selection and classification tests. In theory, a the past score for the new test at the 80th percentile can be said to be equivalent to a score at the 80th percentile on tests, and this equivalence becomes the definition of consistency. ly, it is an advantage if they are parallel, which When several forms of a test are to be operational simultanec allows the use of a single equating table. Gulliksen (1950) offers a definition of parallel tests which includes same- with an ness of factor structure, equality of means, equality of variances, and equality of non-zero correlations external criterion. It also seems reasonable to include equivalence of skew anti kurtosis (Ree, 1977), the third Paid fourth moments of the distribution, although little research exists in the firo'n Parallel tests may be constructed by assigning items randomly to forms. This method is usually called "Ran- domly Parallel Forms." Or items may be matched on difficulty and/or discrimination, stratified, and then assigned random:), to one of a set of multiple forms. This procedure is called "Stratified Parallel." Analytic methods of con- structing parallel forms also exist (Ree, 1976), but they tend to be intensive of computer time. UsinOhe Stratified Parallel method, Forms 8, 9, and 10 of the ASVAB were constructed to be parallel in terms of raw scores so that a single table might be used to convert raw scores on any of the six forms to percentile equivalents. The objective of this study was to determine if a single table were appropriate. Calibration of Tests Because two or more forms of a test can never be made precisely equivalent in range and level, it is necessary (Flanapn, 1951; to render the forms interchangeable by equating. The equating procedure may be defined Angoff, 1971) as converting the scoring units of one test to the scoring units of another. In general, two procedures have been in common use; linear and equipercentile equating. Linear equating requires that equivalent Z-score transformations of the two tests represent the same cumulative proportion. Said differently, the shapes of score distributions should differ only trivially. Equipercentile equating, on the other hand, makes no such assumption of Z-scoke equivalence. The linear method offers the advantage of dealing with analytic statistics (means, standard deviations, etc.) which are verifiable. Equipercentile equating is preferable when the distributions differ and is often offered as the definition of equating (Jaeger, 1981). It should be noted that the linear and equipercentile approach coincide when both the distributions to which they are applied have the same shape. 8 5 example, Angoff (1971) uses the term "calibration" to describe the equating of tests of differing abilities. For it is the equating of a test of Word Knowledge to a test of Reading would be called "calibration." Therefore, Angoff is appropriate to say that military selection and classification tests have beell calibrated rather than equated. calibration. It is somewhat critical of the calibration technique because a problem arises from the nature of lead to repeatedly stated in the literature (Angoff, 1971; Flanagan, 1951; Jaeger, 1981) that calibrating does not the sample-unique solutions, as does equating, although empirical evidence is not offered. The non-uniqueness of of the test. solution makes difficult the, interpretation of several calibrations of the same test, or parallel forms Form 8a of the Military selection and classification tests have frequently been 'alibrated, rather than equated. high school ASVAB was linked via calibration to an anchor test using several differing subject groups ranging from study in order students to new military recruits. The effects of calibrating, as opposed to equating, require further to understand ful!.y the consequences of the technique. 1980) were conducted which Three previous studies (Boldt, 1980; Maier & Grafton, 1981; Sims & Truss, 10 calibrated Form 8a to Armed Forces Qualification Test Form 7a (AFQT-7a). Because ASVAB Forms 8, 9, and previously as "Stratified Parallel Forms," it was reasoned were constructed to be parallel by the method described the that calibrating one form was tantamount to calibrating all forms. That is, because calibration sets raw scores of forms were calibrated test equivalent to raw scores on an anchor or target test, and because the raw scores of the six be applicable to all constructed to be equivalent, then any one form may be calibrated, and the results should then calibrations are the other forms. The crucial requirement is that the forms be parallel. If they are not, separate produced the tables required. The present study seeks to verify the results of the earlier calibration studies which implemented 1 October 1980. These are referred to as the operational tables. ASVAB-8a and thereby In order to determine if the assumptions underlying the procedures for calibrating Evaluation (10T&E) was Forms 8b, 9a, 9b, 10a, and 10b were acceptable, an Initial Operational Test and selection and classification of undertaken. The IOT&E was begun shortly after the test was put into operation for candidates for military enlistment. H. METHOD The Tests subtests. Eight of the Forms 8, 9, and 10 of the ASVAB are multiple aptitude batteries comprised of 10 the number of items, and subtests are power subtests, while two are speeded subtests. Table 1 shows the name, ASVAB forms by the inclusion of whether the subtest is power or speeded. These forms differ from the previous of Automotive Information Paragraph Comprehension (PC) and Coding Speed (CS) subtests, by the combination measuring Space Perception, and Shop Information into a single subtest (AS), and by the deletion of subtests of the forms is about 180 Attention to Detail, and General Information. The overall administration time for any minutes, and in operation, the test is answered on a machine scannable answer sheet. Table 1. Name and Number of Items for Power and Speeded ASVAII Subtests in Forms 8, 9, and 10 Power/Speed Number of Items Name Power 25 General Science (GS) Power 35 Word Knowledge (WK) Power 30 Arithmetic Reasoning (AR) Power 15 Paragraph Comprehension (PC) Speed 50 Numerical Operations (NO) Speed 84 Coding Speed (CS) Power 25 Auto-Shop Information (AS) Power 25 Mathematics Knowledge (MK) Power 25 Mechanical Comprehension (MC) Power 20 Electronics Information (EI) 6 9 The Armed Forces Qualification Test (AFQT) composite is used for military enlistment qualification and is comprised of PC, Word Knowledge (WK), Arithmetic Reasoning (AR), and Numerical Operations (No) subtests. All subtests are unit weighted except for NO, which is weighted by one-half. The AFQT-7a served as the anchor test. This test was previously used for enlistment qualification but has been inactive for several years. It was chosen as the anchor test because its content is close to that of the test used in the 1944 mobilization base development testing. It is not believed to be compromised, and an earlier form (Form 3) of the ASVAB was calibrated against it. The AFQT-7a has 100 items evenly distributed in the ability areas of WK, AR, Boxes (B), and Tool Knowledge (TK). The first two. WK and AR. are similar to the like-named subtests in tlw current ANT portion of the ASVAB. The latter two. B and TK, are not found in the current AFQT portions of the ASVAB. It is the disparity' in the ability nwasured which leads to labeling the equating effort a calibration and which leads to the problem of non-unique solutions. Administration of Tests to Subjects .% sample of subjects was drawn to provide for equal geographical representation. Data collection took place in 20 Armed Forces Examining and Entrance Stations (AFEESs). Table 2 shows the locations of the AFEESs and the number of subjects at each. Each subject took the AFQT-7a and one form of the ASVAB, which was used for qualification for military enlistment. The AFQT-7a was adinin isten.d on a separate answer sheet. The ASVAB arid ANT-7a tests were administered in counterbalanced order by reversing order of their administration each day from that employed the previous day. Tests', were also administered at locations affiliated with the AFEES, called Mobile Examining Team (MET) sites and Office of Personnel Management (OPM) sites. Table 2. AFEES Sites and Sample at Sites' Subjects AFEES 1,500 Chicago 1,300 Cleveland 800 Atlanta 1,600 Baltimore 1,300 Boston 1,400 Jacksonville 2,600 Los Angeles 900 Montgomery 1,400 Newark 1,400 Philadelphia 1,200 Richmond 1,400 St. Louis 500 Spokane 600 Denver 600 Houston Phoenix 500 400 Portland 600 San Diego 1,200 Minneapolis 1,200 Omaha 22,400 Total 3Sites included AFEES, MET, and OPM locations for test administration. 7