. ".DOCUMENT RESUME ED 204 147 TM 007 012 o Pike, Lewis W. AUTHOR TITLE Short-Term Instruction,'Teetwiseness, and the Scholastic Aptitude Test. A Literature Review with Reseirch Recommendations. Educational Testing Servide, -Priffoeton; N.47w4- SPONS AGENCY College Entrance Examination Board, New York, N.Y. PUB DATE 79 47p. NOTE"' AVAILABLE PROM College Board Publication Orders, Box 2815, Princeton, NJ 08541. EDRS PRICE 0MFO1 Plus Postage., PC Not Available from EDRS. DESCRIPTORS *College Entrance Examinations: Guessing (Tests): High Schools: *Instruction *Program Effectiveness: Research Design: *Research Needs: *Scores: *Test. Wiseness: Time Factors (Learning) *Scholastic Aptitude Tept IDENTIFIERS ABSTRACT # 4 The research literature on short-term-instruction (STI) and intermediate-term instruction (ITI) for the Scholastic Aptitude Test-mathematical section :(SATM) and the Scholastic Aptitude Test-verbal sections (SATV1 was reviewed. Selected studies of STI and ITI for tests other than the SAT-M and TV, and of- testwiseness (Me weie included in the survey if they were judged relevant to .the question of special instrlction'for the SAT. The research studies were reviewed and interpreted within the framework' of a score components model that prksited four content-related and two ap TV scores components, as well as test-taking confidence and efficiendy, that are theoretically subject to STI and ITS effects. In addition, examinee, item, and instructional cha#acteristics were considered as they relite tp the score components aodel.°Basic discrepancies between negative and positive findings were noted for ,both the SAT-M and the SAT-V. There "were generally resolved in favor of .recognizing meaningful STI effects for the SAT-M, but remain unresolved- for the SAT-V. Recosuendatiotts vere made for SAT-M and SAT-V research allowing STI effects to be partitioned according to etaminee, item, and instructional characteristics as they apply to 2 selected test score components. (Author /RL) I *i********************************************************************* -Reproductions supplied by EDRS are the best that can be made *' . * from the original document. *4******************************************************************** College Board Research and Development Report Short-Term Instruction, TeshAtiseness, and the Scholastic `Aptitude test tt ( * THIS U.S. DEPARTMENT Of EDUCATION "PERMISSION TO REPRODUCE ONLY NATIONAL INSTITUTE OF EDUCATION MATERIAL IN MICROFICHE IOU TIONAL RESOURCES INFORMATION BY HAS BEEN GRANTED CENTER IERIC) , Ibis' dxurnent by been reproduced as received, horn the person or onsenitetion originsung it 0 Pala clumps Awe been made to improve reproduction (twiny. RESOURCES. TO THE EDUCATIONAL Points& View orcesinionsstatedb this deco. (ERIC)? INFORMATION CENTER ment do not necesserk represent Oho ME Position or petty. 2 ti Short-Term Instruction, Testwiseness, and the Scholastic Aptitude Test A Literature Review with Research Recommendations Lewis W. Piked r College Entrance Examination Board, Newl'ork, 1979 3 7 O The College Board is a nonprofit vembership organization that provides tests and other educational services fon students; schools, and colleges. The membership is composed of more than 2,500'colleges, schools, scho.11 a Representatives of the members systems, and education associations. . serve on the Board of Trustees and advisory councils and committees that consider the programs of the College Board and participate in the determination of its policies and activities. Copies of this report may be ordered from: College Board Publication Orders Box 2815, Princeton, New Jersey 08541. The price Editorial inquiries abouC this report should be addressed to: , The College Board . 888 Seventh Avenue, New York, New York 10019. This book, publ:sLed by the College Board,, is based on wotk commissioned and funded by the Board. All rights reserved. .Copyright 40 1979 by Educational Testing Service. Printed in the United States of Amvica. ". aV 1 i0 Contents Abstract Introduction 5 . Score of the report 5 Components of.observed tgst scores 5 .Some definitions and conceptualizatiens 6- 4 Literature Review 9 . , Instruction for the SAT-Mathematical 9. r Instruction for the SAT-Verba 12 14. Instruction for tests other than the.SAT Studies examining TW 15 ; Summary and Interpretation of Findings 20' . :, 'e General considerations 20 . . . , ... Findings regarding the SAT-14 23 Findings regarding the SAT-V 27 .' 4 , . . Findings,regarding TW 31 . Recdmmendations for Future Research 34 Research objectives 34, Desigh concideratight 37 . . Recommendations for SAT-14 research A . Recommendations for SAT-V research 39 References 41 . Studies of STI and ITI 41 Studds of TW 42 tv 4 cf. C . It .% 0 The research on which this report is based'was conducted-. in 1977 by theauthor. It is being published now; to share Lewis W. Pike's . conclusions with other interested researchers. Stephen W. Ivan, The College Board . A 1 G O Ale 4 .0 C O - .1 fr O Abstract 'The research literature or; short-term instruction (STI) and intermediate- term instruction (ITI) for the SAT-mathematical sections and SAT-verbal sections was reviewed. Selected studies of STI and ITI for tests other - than the SAT-M and SAT-V, and of testwiseness (TW), were included in the survey if they were judged relevant to the question of special instruction for the SAT. The research studies wire reviewed and interpreted within the frameWork of a score cclaponeftts model that posited four content-related and two TW score components; as well as test-taking confidence and efficiency, that are'theoreacally subject to STI and ITr.effects. In addition, e*amtn#e, item, and instructOnal characteristics were considered as they relat e to the score components model. .4asic discrepancies between negative and positive findings were noted ft:ix' both the SAT -M, and the SAT-V. Thve were generally resolved in favor.of recognizing meaningful STI effects for the SAT-M, but remain unresolved for the SAT-T. Recommendations were made for SAT-M and SAT-V research allowing STI effects to be partitioned according to examinee, item, and instructional characteristics as they apply to selected test score components. O 7 3 4, 3 VT, Introduction c , This study was requested by the Collegf Board to provide an up-to-date' summary of research findings' relevant to the question of special Insekuv- tion for the Scholastic Atitude Test. The-need for such a review lies both in die continued relevance of the question, and iti the fact that the last summary was completed several ago (College Board,. 1968). _ for the SAT remain relevant for Question's regarding special ins ruction several reasons. One reason is that the continued importance ofSAT scores to examinees results in a continued pressure to obtain "instruction for the SAT," whichin turn leads to an actAve commercial "coaching" enterprise and to efforts by some public and private.schools to prAide such instruc- Lion. Another is that the changing make-up of pile examinee population needs examination. It is entirely iiausible,. forexample, that the more advantaged students represented in most of the studies cited in the College Board booklet, Effects -of Coaching on ScholastiokAptitude Test Scores (most of them conducted in tfie1950s), were already well prepaied to d their best proportion on the SAT.to a degree that cannot be assumed for an increasi ? of the current candidate population, particularly minority an other stu- ' dents outside the mainstream of educational opportunity. Finally, there have been studies of instruction directed either to the SAT or to closely related tppicssuch as the "coachability" of 'verbal analogies that have appeared since the College, Board booklet was published that need to be con- sidered in current thinking and general statements regarding instruction for*the SAT. .. , . 4 SCOPE OF THE REPORT The literature review will cover two interrelated areas of study: (1) studs' ies of short -tern instruction (STI) and intermediate-term instruction (YTI) directed specifically toward increasing test scores, with particular empha- _sis_on_the SAT-Ar and SAT-14 and (2) studies-ortestwiseness-(110------1---- that were not specifically directed to raising test scores. The review of- the literature will be followed by recommendations for future research., Two. topics will be considered nextIthat should help clarify the sub- sequent review of the literature and facilitate the discussion of its im- plications: the components of observed test scores as theyarelate to questions of short-term instruction (STI) and testwiseness (TW); and 'definition of terms. , COMPONENTS Op OBSERVED TEST SCORES 2 Implicit in many discussions of STI and TW is the assumption that an in- dividual's test score is essentially a composite of the ability or 5 8 ,;% knowledge forwhich a person is being testea,.testwiseness, "error"-chance factors in the' sampling of test items--lucky guessesarid so on. This assumpion is often accompanied by the belief-that the intended "real" or V "true" score on aptitude tests such as the SAT-rand SATM is necessarily (by definitiOn) subject only to gradual, 19ng-term change and, as a . corollary, a distrust or suspicion of anything.that-might alter aptitude , test scores in a relatively short term (i.e., STI). To put this question in perspective, it is useful to consider the following delineation of the components of observed test scores. *. A. "True score" components: e.g., verbal aptitude, mathematical aptitudt. 1. A composite of underlying knowledge (e.g., vocabulary, elementary algebra) and reasoning ability, developed over a long period of time. Long -term acquisition, long-term retention.) /. A state of being well-r6iewed, so that the performance to be demon- strated is in line, with the individual's underlying developed competence. (Short-term acquisition,, short- or mediUm-term retention:) 3: Integrative learning, overlearning, contolidation. (Short-term. acquisition, long-term retention.) 4. Learning criterion- relevant, analytic skills -(e.g., how to identify the main idea of a paragraph; how to.simplify complex quantitative terms before compaiing their value). (Short-term acquisition, long- term retention.). B. Primary test-siecific components. 1. The match between developed abikitys(including the various score compbnents listed in A above) and, test content. Mismatches may occur as gaps in such areas as skill in locating information in reading passages and ability to work with the algebra of inequalities. 2. General TWtest familiarity, pacing, understanding of general direc- tions, general strategies for using partial information, and so on. 3. Specific TWcomponents similar to B2, but in reference to charac- teristics of specific itcm formats-(such ag verbal analogies and quantitative-comparison items)', and other item characteristics. C. Secondary components influencing test taking. 1. Level of confidence. 2. Level of efficiency- -the ability-to use available knowledge and reasoning ability' quickly with'a relatively lbw rate of error re- sulting from,vorkinirepidly. D. "Error."'Fluctuations in attention, sampling error, variations in luck when guessing, etc. SOME' DEFINITIONS AND CONCEPTUALIZATIONS the "aptitpde!_yersus Terms such as STI, ITI, Coaching, TW, guessing:\add _ achievement"diseirictionire central to 'discussions regarding special preparation for test taking, and the meanings of these terms tend to vary . from one writer to the next. It will be useful, therefore, to give a brlef definition of each, as used in this review, and to expand on the conceptu- alizations where needed. Short-term instruction (STI). The term STI will refer to attempts to improve test scores by means of a relatively short period of instruction; relatively'short, that is, when compared to the amount of time generally, considered necessary for any substantial change in the ability or knowledge in question. STI may, be directed toward any or all of the components of observed test scores noted above except.true-score component Al, which is by definition limited to long-term acquisition. Note that STI for compo- 0. Y nents A2, A4 and A4 'is il_fact directed toad the ability of .interest, thtugtjle iustructron is short-term. It ray'ke added that ip.general even there is no sharp contrast between educ4kion and STI; given appropriate contain,/ ST;,may prOperly be viewed'as instruction provided in addition, ' to, rater than instead of, 'conyentiona/Aong-term Learning (i.e., componint- _ . . Intermediate-teim instruction (ITI). As the name suggests, ITI will refer to attempts to improve'test scores by means of special instruction for a '6 somewhat longer'period than STI but still a short period compared to the amount of time generally, considered necessary for substantial changes in the ability in question. Mecept for the difference in the relative period of instruct .oh, the description given of STI also applies to ITI. This term will refer to a subset of possible STI activities ,Coachins limited essentially to very brief instructionin general testwiseness, such -as effective pacing, answering items whenever partial information about them is known, and practice in answering question's similar to those in the 'target examination. Specifically not included in this definition of coach- ing is any content instruction beyond that which is merely incidental to the practice sessions. This definition is implicit in the College Board (1968) statement on coaching,'in the design of most if the studies re- ported there, and in-the interpretation of their results. It has been fairly widely adopted, as is indicated in a recent statement on coaching made py Anastasi;(1976): "Item types on which performance can be appre- ciably raised by short-term drill or instruction of a narrowly_limited nature are not included in the operaxiodal forms of the (SAT) tests" (p.4.43) . Tesmiseness (TW). In essence, TW is a set of skills and knowledge about test taking that enables individuals to display their abilities (e.g., verbal and mathematical aptitude) to their test advantage. A TW component is by no means unique to standardized tests. It is also.present in other - modes of assessment such as classropm recitation and essay writing. Early recognition of the TW component in SAT scores is evident from the fact that "From 61926 to 1944 candidates were required'w present completed, practice booklets before they were allowed to take.the test" (Fremer and Chaedler, 1971. p. 147). 2114 InStruction'is sometimes viewed primarily as an effort eo beat the test, with the assumpift that testwise examinees will somehow get higher scores than they deserve. For well-made standard- ized tests, however, clues that offer spurious routes to correct answers % are scrupulously avoided, 'and the opposite, mole compelling concern is that examinees who are not testwise may receive inappropriately low scores. Thus, Stanley (1971, p. 364) uses the contrasting term "test-naivete," aftd Ebel (1'65) notes that "More eiror in measurement is likely to orAginape from the students who have had too little, rather than too much, skill in to 'sing tests" 206). k. Guessing. Stated simply, guessing consists of answering a test question in the absence of certainty as to the correct response. It may be divided into three categories; 'guessing that is blind .or random, guessing that is spurious or based on a hunch, and guessing based on partial information. In contradistinction to the common feeling that guessing is at'least faintly disreputable, the following out points should,be noted. First, guessing is necessary for responding appropriately to the SAT and to most kinds of assessment. Most examinees encounter some test questions about which they have partial information that would enable them to elim- inate at least one choice. In such cases they must guess among the remain- ing alternatives if they are to benefit from their partial information. ' 7