Different Tests, Same Flaws: Examining the SAT I, SAT II, and ACT by Christina Perez Spurred in part by University of California (UC) President Richard Atkinson’s February 2001 proposal to drop the SAT I for UC applicants, more attention is being paid to other tests such as the SAT II and ACT. Proponents of these alternative exams argue that the SAT I is primarily an aptitude test measuring some vague concept of “inherent ability,” while the SAT II and ACT are “achievement tests” more closely tied to what students learn in high school. While the origins of the exams and the rhetoric of their promoters may differ, the SAT I, SAT II and ACT actually exhibit many of the same flaws and shortcomings. Are Admission Exams Really that Different from One Another? The roots of the SAT I lie in the racist eugenics movement of Christina Perez is the university testing reform advocate the 1920s. Carl Brigham, a psychometrician who worked on for FairTest (http://www.fairtest.org). Her work focuses the Army Alpha Test (an “IQ” test administered to World on encouraging colleges and universities to drop the use of SAT and ACT scores in admission decisions, as well War I recruits that was used to justify racial sorting), was as promoting other civil rights issues associated with hired by ETS in 1925 to develop an intelligence test for standardized tests and higher education. She previously college admission. At that time Brigham believed that people worked as a research associate at TERC in Cambridge, of different races could be rank ordered by intelligence level, MA, where she assisted math curriculum writers and with African Americans at the lowest point in the pecking workshop developers in increasing the equity-related content of their workshops for teachers. She holds a dual order (Lemann 1999). Drawing from his work on the Army bachelor’s degree in public policy and women’s studies Alpha Test, Brigham designed the earliest version of the SAT I, from Brown University (RI). which was at that time taken by just over 8,000 students and promoted to elite colleges primarily in the Northeast. Al- 20 T H E J O U R N A L O F C O L L E G E A D M I S S I O N The intended purpose of the SAT I, SAT II, and ACT is to help predict first- year college grades. Yet high school grades, class rank, and rigor of courses though it included some mathematical calculations and identification of shapes, this early version of the SAT was do a better job of forecast- focused on word familiarity—a question format popular in intelligence testing. While the SAT I has been modified over the ing college performance years, the current test remains quite similar in format and content to Brigham’s 1925 creation. than any of the tests. Several years earlier, the College Board began administer- ing the College Entrance Examination Board Achievement Tests. Covering subjects such as English, French, Greek, into effect in March 2005 and will be accompanied by a new mathematics, botany, and physics, the initial exams were combined high score of 2400 (the current perfect SAT I score primarily composed of essays that tested mostly rote memori- is 1600). Although these changes are being promoted as a zation of discrete facts. Criticism from high schools, charging move toward making the exam more “achievement” based, that the board was trying to dictate what they should teach, this is largely a marketing strategy in response to criticism led to revisions in the entrance exams (Crouse and Trusheim about the test from places such as the University of California. 1988). In 1916, the College Board offered new comprehensive Most of the test’s format and question content will remain exams that covered a wider range of material and were used unchanged, as will its disconnection from actual classroom exclusively by highly selective institutions of higher education. learning. These were renamed the SAT II: Subject Tests in the mid-1990s In 1959 ACT, Inc. came into the picture as the manufac- and matched to an existing 200–800 point scale. turer of a third major admission test. The ACT, created by E.F. Today the College Board tries to distinguish between the Lindquist (who also designed the Iowa Test of Basic Skills) and two exams by claiming that each measures something unique: Ted McCarrel, was initially intended to compete with the SAT I’s foothold in college admission. According to the test’s The SAT [I] measures verbal and mathematical reasoning manufacturer, the ACT is distinguished from the SAT I by its abilities that students develop over time, both in and out link to classroom curriculum: “The ACT Assessment tests are of school, which are related to successful performance in curriculum based. The ACT Assessment is not an aptitude or college…SAT II: Subject Tests are designed to measure an IQ test. Instead, the questions on the ACT are directly knowledge, and the ability to apply that knowledge, in related to what is learned in high school courses in English, specific subject areas. Students take the SAT II: Subject mathematics, and science” (ACT, Inc. Web site 2002). Students Tests to demonstrate to colleges their mastery of specific are rated with four subscores (Reading, English, Mathematics, subjects.” (College Board Web site 2002) and Science Reasoning), as well as a Composite score (1-36 scale). However, the Board’s position on what the two exams In fact, students’ scores on all three exams tend to be very measure is likely to shift given the changes to the SAT I similar. According to the College Board’s own research the announced in June 2002. In an attempt to make the SAT I correlation between SAT I and SAT II scores hovers around more closely tied to high school curriculum, the verbal analo- .84, while the correlation between SAT I and ACT scores gies section will be replaced with short reading comprehension ranges from .89 to .92 (College Entrance Examination Board passages; Algebra II will be added to the math section; and a 1999a). In other words, students receive very similar scores on new “writing” section will be included that is based on the each of the exams, calling into question how reliable test- current SAT II Writing Test. The changes are expected to go maker claims are that the exams measure something unique. Indeed, both the College Board and ACT produce concordance tables which allow the translation of SAT scores into ACT equivalents, and vice versa. F A L L 2 0 0 2 21 One reason for the strong relationship among SAT I, SAT II and ACT scores may be their similarity in format. All three are timed, multiple-choice tests normed on national samples of students. Since students are tested under similar conditions and in a similar way, the exams tap a narrow range of skills and cater to one kind of learning and test-taking style. If a student does not do well at timed, multiple-choice exams, his or her true abilities may not be reflected in the final score. For example, research has shown that the “speeded” nature of the How Well Do Admission Exams Predict Success in College? exams tends to put females and students whose first language The intended purpose of the SAT I, SAT II, and ACT is to help isn’t English at a disadvantage (Kessel & Linn 1996; Schmitt, predict first-year college grades. Yet high school grades, class et al 1991). Such research may explain why females earn rank, and rigor of courses do a better job of forecasting higher grades in high school and college but receive lower SAT college performance than any of the tests. College Board and ACT scores. claims that the SAT I explains approximately 27 percent of the Similarity in test question content is another explanation variation in first-year grades, while SAT II exams on average for the high correlation between scores. Items on the SAT I- explain around 16 percent of the differences (College Entrance Verbal, SAT II Literature, and ACT Reading tests all ask Examination Board 1999b; Ramist et al 2001). Although the students to read a short essay and answer multiple-choice addition of a third component on the SAT I may slightly questions based on the passage’s content. The SAT II Writing improve the test’s predictive power, the change will likely be (multiple-choice section) and ACT Reading tests both contain small at best and will not be equal across demographic groups. English “grammar” items in which test-takers need to select The ACT has a similar predictive validity: 17 percent of the correct words or phrases to complete a sentence. A similar explained variance (ACT 1998). High school grades—the component will also be added to the “new” SAT I, making the single best predictor of first-year performance according to three tests resemble one another even more than they cur- both test manufacturers—account for nearly 30 percent of the rently do. For example, a SAT II Writing question asks test- difference in freshman grades, even in an era of grade inflation takers to indicate which, if any, of the underlined words or and variability in quality among high schools. phrases contains a grammatical error: Independent studies confirm these findings. The Case Alexis has discovered that she can express her creativity Against the SAT by James Crouse and Dale Trusheim demon- more freely through her sketches and not in her photography. strates the SAT I’s poor utility in forecasting both short- and No error. (College Entrance Examination Board 1994) long-term academic success. The authors compared two On a similar ACT question, test-takers must indicate the admission strategies, one using just the high school record and correct word choice for the following sentence: the other using high school record and SAT I scores. More Japanese students observe a rigorous annually schedule. than 90 percent of the admission decisions were the same (ACT 1997) under both strategies. However, for the 10 percent of the The answer choices consist of the following: a) no change, applicant pool in which the two strategies led to different b) an annual rigorously, c) an annual rigorous, or d) a admission decisions, the SAT-based approach led to a far rigorous annual. greater number of rejections of otherwise academically The SAT I-Math, ACT Mathematics, and SAT II Math qualified African-American and low-income applicants IC tests cover very similar content and format (although the (Crouse and Trusheim 1988). SAT I does not include trigonometry or statistics). Sample questions from each of the three exams illustrate this point (College Entrance Examination Board 1994 and 1995; ACT 1997): SAT I SAT II ACT If (y + 2)2 = (y – 2)2, If f(x) + 1 – x3, then f(-1) = If (x + k)2 = x2 + 22x + k2 what is the value of y? for real numbers x, then k = ? a) 0 b) 1 c) 2 a) –2 b) –1 c) 0 a) 11 b) 22 c) 44 d) 4 e) 6 d) 1 e) 6 d) 88 e) 176 Test-maker claims about the radically different content and purposes of the exams do not hold up upon a comparison of test questions. 22 T H E J O U R N A L O F C O L L E G E A D M I S S I O N The new writing section of the SAT I will not alleviate that exam’s “coachability,” Researchers at the University of Pennsylvania looked at the power of high school class rank and SAT I and SAT II and may , in fact, increase scores in predicting cumulative college grade point average (GPA). They found that the SAT I was by far the weakest the ability of test prep predictor, explaining only 4 percent of the variation in college grades, while SAT II scores accounted for 6.8 percent of the courses to boost students’ differences in academic performance (Baron and Norman 1992). By far the most useful tool proved to be class rank, scores. which predicted 9.3 percent of the variation in cumulative GPAs. Combining SAT I scores and class rank inched this The Impact of Test Coaching figure up to 11.3 percent, still leaving almost 90 percent of the Further diminishing their power to predict academic perfor- variation in grades unexplained. mance in college, the SAT I, SAT II, and ACT are all suscep- Chicago State University researchers looked at the ability tible to coaching. While there is a great deal of debate sur- of the ACT to predict college academic performance. For the rounding how coachable exams are, numerous studies have vast majority of the university’s graduates who had mid-range demonstrated that student scores on admission exams can be ACT scores, the test explained only 3.6 percent of the differ- significantly boosted through rigorous coaching. For example, ences in cumulative college GPA (Paszczyk 1994). In fact, the the Princeton Review test preparation company guarantees a exam over-predicted the performance of students graduating in 100+ point increase in SAT I scores, and a 4+ point increase in 1992, who had the highest average ACT score among the ACT scores (equivalent to approximately 150 points on the classes in the research study yet the poorest academic perfor- SAT I) for students enrolling in one of their test prep classes. mance over four years at the university. The predictability of the exams’ formats and their narrow Research on the exams’ ability to foretell long-term range of content are cited by test prep companies as the success shows even weaker predictive power. Crouse and primary reasons why students’ scores can be boosted through Trusheim demonstrated that using the high school record alone coaching. The Princeton Review Web site states: to forecast who would complete a bachelor’s degree resulted in The ACT tests the same information the same way, year “correct” admission decisions 73.4 percent of the time, while after year…By reviewing the very specific knowledge that using the SAT I and high school GPA was accurate in 72.2 the people who write the ACT think is important, and by percent of the cases (Crouse and Trusheim 1988). learning good test-taking strategies, it should be possible Drawing from a national database of nearly 10,000 to increase your ACT score significantly” (2002). students, one study sponsored by the U.S. Department of Kaplan, the largest test prep company, makes similar Education considered the value of high school grades, class claims about the SAT I: “The SAT I lends itself to preparation rank, test scores, and rigor of courses in predicting attainment because it is a predictable test concentrating on a set of specific of a bachelor’s degree. High school curriculum offered a higher reasoning skills that can be learned.” (Kaplan Web site 2002). correlation with bachelor’s degree attainment (.54) than either The SAT II exams are equally susceptible to coaching. David SAT scores (.48) or class rank/GPA (.44) (Adelman 1999). Owen writes in None of the Above: “As John Katzman Significantly, high school curriculum was an even stronger [president of the Princeton Review] said…ETS’ Achievement predictor for African-American and Latino students than for Tests [SAT IIs] are even easier to coach than the SAT [I]—not students overall, indicating the value of employing broader because, as some have said, their subject matter is more finite admission criteria than just a test score/high school GPA index but because the method by which they are put together is more when trying to forecast the college performance of transparent” (Owen 1999). The new writing section of the underrepresented minorities. Researcher Clifford Adelman SAT I will not alleviate that exam’s “coachability,” and may, in concluded: “[T]he intensity and quality of curriculum is a fact, increase the ability of test prep courses to boost students’ cumulative investment of years of effort by schools, teachers, scores. The narrow criteria used to grade essays lends itself to and students, and provides momentum into higher education coaching, as evidenced by the success test prep companies have and beyond. It obviously pays off. The effects of grades and in preparing students for the SAT II Writing Test. tests diminish in time, but the stuff of learning does not go Score gains that come with test coaching exacerbate the away.” inequities already present with college admission exams. These courses, which can cost $800 or more, skew scores in favor of higher-income test-takers, who already tend to do well on the exam (Stockwell et al 1991). Because college admission officials do not know who has been coached and who has not, they cannot fairly compare two applicants’ scores. F A L L 2 0 0 2 23 Will Substituting One Exam for Another Increase Racial Diversity? Some critics of UC President Atkinson’s proposal to drop the SAT I and rely instead on tests of “academic achievement” such as the SAT II or ACT argue that the measure is simply a way to skirt the state’s ban on affirmative action. Yet similar racial/ethnic score gaps exist on the SAT II and ACT. The following chart illustrates the deep divides between students of different ethnic groups on all three admission exams: Ethnic Group Score Gaps (Points above or below White Average) SAT I Verbal + Math SAT II Writing SAT II Math IIC ACT Composite (SAT equivalent) African American -198 African American -82 African American -69 African American -4.9 (-197) American Indian -95 American Indian -51 American Indian -37 American Indian -3.0 (-120) Asian/Asian American +6 Asian/Asian American -30 Asian/Asian American +16 Asian/Asian American -0.1 (-4) Mexican American -145 Mexican American -106 Mexican American -63 Mexican American -3.3 (-132) Other Latino -130 Puerto Rican -151 Puerto Rican -67 Puerto Rican -39 Puerto Rican -2.4 (-100) Source: College Entrance Examination Board, 2001 College-Bound Seniors: A Profile of SAT Program Test-takers and written correspondence with the College Board; ACT, Inc., The High School Profile Report Normative Data 2001. The score gaps on the SAT I and ACT Composite are almost identical and will not likely change with the introduc- tion of the “new” SAT I in March 2005. Many colleges that require SAT II exams ask applicants to submit scores on three Is There a Better Way? or more subject tests. When the score differences on the most Despite differences in test-maker rhetoric, the SAT I, SAT II, commonly required SAT II Writing and Math IIC are added and ACT contain similar flaws and shortcomings. All three together with a third exam (most of which have score gaps exams have a weak ability to predict academic performance in similar to those on the Writing and Math tests), the gaps will college, while high school grades, class rank, and rigor of match, if not exceed, those present with the SAT I and ACT. In classes taken are better predictors of success. All three exams other words, replacing the “old” SAT I with the “new” SAT I are highly coachable, advantaging students who can afford to or with the ACT or multiple SAT II exams will do little to spend $800 or more on test preparation classes. All three increase racial diversity on college campuses if test scores are exams have a similar format, placing groups such as females heavily emphasized. and English as a Second Language learners at a disadvantage since they do not tend to perform as well on timed, multiple- choice exams. All three exams show large gaps in scores All three exams have a weak between students of different racial and economic groups, leading to racial and class bias in admission and financial aid ability to predict academic formulas that utilize rigid test score requirements. Finally, all three exams place the financial and time burden on students performance in college, rather than universities, making them low-investment sources of information for colleges but high-investment hurdles for while high school grades, students. class rank, and rigor of classes taken are better predictors of success. 24 T H E J O U R N A L O F C O L L E G E A D M I S S I O N These findings lead to one central conclusion: substituting one university References admission exam for another ACT, Inc. 1998. ACT Prediction Research Summary Tables. Iowa City, IA: ACT, Inc. benefits neither students Crouse, J., and Trusheim, D. 1988. ACT, Inc. 1997. Getting Into the The Case Against the SAT. nor schools. Colleges that ACT: Official Guide to the ACT Chicago: The University of Assessment. San Diego: Harcourt Chicago Press. are genuinely committed to Brace & Company. Kaplan, Inc. 2002. Web site: ACT, Inc. 2002. Web site: increasing the representa- http://www.kaplan.com. http://www.act.org/aap/faq/ general.html. Kessel, C. & Linn, M. 1996. tion of African Americans, “Grades or Scores: Predicting Adelman, C. 1999. Answers in the Future College Mathematics Tool Box: Academic Intensity, Latinos, and Native Ameri- Performance,” Educational Attendance Patterns, and Measurement: Issues and Bachelor’s Degree Attainment. Practice, Vol. 15, N.4, Winter. cans must look into other Washington, DC: U.S. Depart- ment of Education. Owen, D. with M. Doerr. 1999. alternatives. None of the Above: The Truth Baron, J. and Norman, M.F. 1992. Behind the SATs. New York: SATs, Achievement Tests, and These findings lead to one central conclusion: substituting High School Class Rank as Rowman & Littlefield Publish- ers, Inc. one university admission exam for another benefits neither Predictors of College Perfor- students nor schools. Colleges that are genuinely committed to mance. Educational and Paszczyk, S. 1994. A Comparative Psychology Measurement 52: increasing the representation of African Americans, Latinos, Analysis of ACT Scores and 1047-1055. and Native Americans must look into other alternatives. Final GPA’s of Chicago State Test-score optional policies are one significant step toward College Board. 2002. Web site. University Undergraduate Students. ERIC Document no. a sound and equitable admission process. Nearly 400 bachelor- http://www.collegeboard.com/ 370 519. degree granting institutions now admit a substantial number of sat/satguide/html/satinfo.html. freshman applicants without regard to SAT or ACT scores (see Princeton Review. 2002. Web site: College Entrance Examination http://www.fairtest.org/univ/optional.htm for a list). Regardless http://www.review.com. Board. 1994. The Official Guide of size, selectivity, geography, and student body composition, to SAT II: Subject Tests. New Ramist, L., C. Lewis, and L. “test-score optional” institutions report widespread success in York: College Entrance McCamley-Jenkins. 2001. Using enrolling talented pools of diverse students. Examination Board. Achievement Tests/SA- II: The debate about which test score(s) to require doesn’t Subject Tests to Demonstrate College Entrance Examination acknowledge the larger issue present in reassessing college Achievement and Predict College Board. 1995. Real SATs. New admission practices—all current standardized admission exams York: College Entrance Grades: Sex, Language, Ethnic, and Parental Education Groups. act as gatekeepers to students of color and low-income stu- Examination Board. New York: College Entrance dents. Until university officials and test-makers are willing to Examination Board. College Entrance Examination confront this fact, shifting admission policies from one test Board. 1999a. Concordance score requirement to another will simply uphold the faulty Schmitt, A.P., Dorans, N.J., Crone, Between SAT I and ACT Scores paradigm that “test scores equal merit” and maintain the For Individual Students. New C.R., & Maneckshana, B.T., 1991. Differential Speededness narrow pipeline through which traditionally underrepresented York: College Entrance and Item Omit Patterns on the groups struggle to pass. Examination Board. SAT. Research Report 91-50, Princeton, NJ: Educational College Entrance Examination Testing Service. Board. 1999b. Counselor’s Handbook for the SAT Program Stockwell, S., R. Schaeffer, and J. 1999-2000. New York: College Lowenstein. 1991. The SAT Entrance Examination Board Coaching Cover-Up, Cambridge, National Center for Fair & Open Testing. F A L L 2 0 0 2 25