o ... DOCUMENT RESUME HE 015 044 ED 215 653 'Barrows, Thomas S.; And Others AUTHOR College Students' Knowledge and Beliefs: A Survey of TITLE-' Global Understanding. The Final Report of the Global Understanding Project. Council on Learning, New Rochelle, N.Y.; Educational INSTITUTION Testing Service, Princeton, N.J. Department of Education, Washington, DC.; EXXON SPONS AGENCY Education Foundation, New York, N.Y.; National Endowment for the Humanities (111FAH), Washington, D.C. REPORT NO ISBN-0-915390-31-0 PUB DATE 81 NOTE 294p. Change Magazine Press, 271 North Avenue, New AVAILABLE FROM Rochelle, NY 10801 ($10.95). MFO1 Plus Postage. PC Not Available from EDRS. EDRS PRICE Area Studies; *College Students; Cross Cultural DESCRIPTORS Studies; *Cultural Awareness; Data Collection; Foreign Policy,: *Global Approach. Higher _Education; International Relations; Knowledge Level; Language Proficiency; *National Surveys; Research Design; Research Methodology; Second Language Learning; Student Attitudes; *Student Evaluation; Tests; *World Affairs ABSTRACT The development,,, administration, and evaluation of a national survey to determine college students' understanding of world affairs are described in 12 articles that focus on survey measures, procedures, and results. Stephen F. Klein and Sheila M. Ager describe the issues examined by an assessment committee, their choice of an issues framework for the test, and the selection of content areas, including foreign affairs, world history, and area studies. Thomas S. Barrows discusses four types of measurement methods that were employed: Likert, self-report, semantic differential, and error choice. John L. D. Clark discusses the rationale for the foreign language component, data selection instruments;, pretesting of self-appraisal technique, and language pretest results. Mary F. Bennett briefly addresses elements of students' backgrounds that might influence the development of global understanding. Henry I. Braun considers stratification, sample selection, sample characteristiCs estimationand variance estimation. Lois G. Harris, MarT. Bennett, and Thomas S. Barrows-examine recruitment, _ administration, and sampling error, and Stephen F. Klein and Sheila M. Ager discuss structure of the knowledge domain, level of test performance -, -and item level performance. Mary F. Bennett highlights data that are relevant from-foreign language background, proficiency, and attitude standpoints. Thomas S. Barreit-presents-the_results of the surveys, and Thomas S. Barrows and John L. D. Clark report on analyses to fit together response data and survey measures. Additionally, Thomas S. Barrows provides a summary and interpretations. Questionnaires and a list of participating colleges are appended. (SW- '"--""1 ...... 0 ... Educational Testing Service Princeton, New Jersey College *dents' Knowledge and Beliefs: __. .., A Survey of Global Understanding ... The Final Report "PERMISSION to REPRODUCE THIS MATERIAL IN MICROFICHE of the ONLY U.S. DEPARTMENT OF EDUCATION HAS BEEN GRANTED BY NATIONAL INSTITUTE OF EDUCATION Global Understanding Project t EbucA *NAL RESOURCES INFORMATION CENTER 1E1110 L., Thts document has been reproduced as received from the person or organizetan originating it. TO THE EDUCATIONAL t2; Minor changes have been made to improve RESOURCES by INFORMATION CENTER reproduction quality. (ERIC)." Points of view or opinions stated in this docu- ment do not necessanIy represent official NIE Thomas S. Barrows posmon or policy Project Director with contributions by Sheila M. Ager, Mary F. Bennett, Henry I. Bratin, John L. D. Clark, Lois G. Harris and Stephen F. Klein C°1-. _1981 Change Magazine Press .. . rl 2 _ April 1981 This volume is the complete technical report of the 1980 Global Un- derstanding Studerit Survey conducted by the Educational Testing Service of Princeton, New Jersey. This report and a series of related Council on Learning publications result from the Council's Education and tie World View Project. Funding for the ETS survey of students' global understanding was provided by the Office of International 'Education of the U.S. Department of Education, with the support of the Natiimal Endowment for the Humanities. Funding for the total Council on Learning project was provided by the National Endow- ment for the Humanities and the Exxon Education Foundation. Ad- ditional copies of this repot t, as well as related publications listed at the end of this volume, can be ordered from Clu.,ge Magazine Press, 271 North Avenue, New Rochelle, N.Y. 10801. Copyright © 1981 Change Magazine Press 271 North Avenue New Rochelle, N.Y. 10n1 ISBN 0.915390 -31-0 LC M80-69767 $10.95 each .. Contents . Foreword i Acknowledgments ii Introduction iii Part I: Survey Measures 1 Development of the Test of Knowledge 1. 2 Stephen F. Klein and Sheila M. Ager The Assessment Committee, 3 Structuring the Issues Framework, 4 Attitudes and Perceptions 2. 7 Thomas S. Barrows 'Liken. Scales, 7 Self-Report Scales, 17 . Semantic Differentials, 17 Error Choice, 23 Language 3. 25 John L. D. Clark Rationale for Inclusion of Foreign Language Component, 25 Consideration in. Selection of Data Selection Instruments, 26 Pretesting of Self-Appraisal Technique, 27 Language Pretest Results, 31 Background and Interests 4. 36 Mary F. Bennett Part II: Survey Procedures 39 Survey Design 5. 40 Henry I. Braun Stratification, 40 Allocation within strata, 43 Allocation between strata, 43 Sample Selection, 44 Sample Characteristics Estimation, 47 Variance Estimation, 49 Data Collection 6. 56 Lois G. Harris, Mary F. Bennett, Thomas S. Barrows Recruitment 50 Administration, 50 Illustrative Sampling Error, 52 .{' Part III: Survey Results 55 Knowledge 7. 56 Stephen F. Klein and Sheila M. Ager Structure of the Domain, 56 Level of 'lest Performance, 60 Item Level Performance, 65 General Background 8. 78 Mary F. Bennett Language 87 9. John L., D. Clark Background Vriables, 87 Proficiency Variables, 90 Attitude Variables, 94 Attitudes and Perceptions 10. 101 Thomas S. Barrows Student Opinion Survey, 101 Student Self-Perceptions, 106 Assessment of World Issues, 108 Error Choice, 114 Correlates of Attitudes, 117 Background/Affect, 117 Experience/Affect, 117 Politics/Affect, 121 Information-Media/Affect, 121 Language Background and Attitudes/Affect, 121 11. 'Structure 123 Thomas S. Barrows and John L. D. Clark Part IV: Summary and Interpretations 133 Global Understanding 134 12. Thomas S. Barrows Appendices A. Booklet A 139 Booklet B 201 B. Language Pretest Questionnaire 263 C. Error-Choice References D. 272 Sample Forms and Correspondence 273 E. Participating Colleges and Universities 286 F. Bibliography 288 5 Foreword L Suspicion; abound that American higher education, in light of changed world circumstances, does not sufficiently prepare the young for their civic roles as they enter the next century. This new edu- cational challenge has led leaders across all walks of American life to advance the widening of under- standing of these new global realities. On one front, Congressman Paul Simon, who has consistently called public attention to lack of global understanding in the United States, pressed for and ob- tained a presidential commission that reviewed the general state of the nation's foreign language and international studies education. One result from that endeavor was the consensus generated for enacting Senators Robert Stafford and Jacob Javits' proposals integrating international education with improving the United States' international economic position. On another front, the Council on Learning, in its catalytic role of focusing national concern on critical higher learning issues, initiated a significant public action project, Education and the World View (E&WV); to strengthen international dimensions in the college experience. George W. Bonham, executive director of the Council, hid succinctly proffered what many in higher education have been certain reflects actual student !taming in this area: America's young face a set of new national and international circumstances about which they have only the faintest of notions. They ate, globally speaking, blind, deaf, and dumb; and thus handi- capped, they will soon determine the future directions of this nation. To remedy this condition he called for a major review of international experiences at colleges and universities across the country. Before any fundamental changes in the American undergraduate curriculum could be encour- aged, it was necessary to assess the quality of campus programs and the degree to which college stu- dents understood world realities: The Educational Testing Service (ETS) was asked to design and conduct a, national survey of college students' global understanding. (ETS had already broken new ground in this area with its earlier precollegiate study, Other Nations, Other Peoples.) The results of this three-year effort are presented in this report. William W. Turnbull, until re- cently president of ETS, and Winton H. Manning, a senior vice president, have correctly stated that this new survey represents active evaluation in the service of and at the disposal of the entire higher education community. Not destined to gather dust on some shelf, the ETS Global Understanding Project report and other E&WV activities are being followed by regional workshops and individual campus self-assessments, so that international dimensions become strengthened throughout the curriculum. To do this, Ef S is providing components included in an E&WV workshop kit that wii! be the centerpiece of these forthcoming campus self-evaluations. Both the Council on Learning and Educational Testing Service greatly appreciate the widespread support and counsel received for the E&WV Project. Special thanks for funding of various segments goes to the National Endowment for the Humanities, the U.S. °Department of Education, and the Exxon Education Foundation. Without such assistance, the objectives of this critical endeavor could never have been realized. gobert Black Prdject Director Education and the World View Acknowledgments This volume is the final report of a survey of global understanding administered to a nationally representative ° sample of college students during February and March of 1980. Several organizations have been responsible for planning, supporting, and directing the effort. The need for an up-to-date scientific survey was first suggested by the 5-0 prominent American leaders from the academy, business, government, foundations, and media, who constitute the National Advisory Board of the Council on Uarning's Education and the World View (E&WV) Project. The actual status of students' knowledge, attitudes, and abilities to comprehend and deal with a new, emerging international order has not been assessed in any systematic way in recent years and, although the need for curricular reform has been broad- ly recognized, it has been based largely on informal impressions and anecdotal evidence. At the request of the F.&WV "task force, Educational Testing Service prepared a proposal for the desired survey which was subse- quently submitted to the National Endowment for the Humanities for funding under the Council's project "umbrella." The resulting contract (NEH-C-93) ?provided support from the Office of International Education of the U.S. Department of Education and project monitoring and direction from both that office and the Na- tional Endowment. While the contract funded Educational Testing Service directly, the project has remained part of the Council on Learning's E&WV project by common agreement and through the National Endow- ment's coordinating efforts. Educational Testing Service has provided substantial financial support for the pro- Jett in order toillow the scope of the Understanding Project to be expanded beyond th original propos- al's specifications. The support of senior individuals at each of these organizations has been critical to the project. The National Endowment's encouragement and intereks must be specially recognized in the efforts of Joseph D. Duffey, Chairman, and Stanley F. Turesky, Assistant Director for Evaluation and Assessment Studies. Ernest L. Boyer, --formerU.S; Commissioner of Education, now President of the Carnegie Foundation for the Advancement of Teaching, and Robert C. Leestma, former Associate Commissioner for Institutional Development and Interna- , tional Education, provided support and direction; the continuing concern of Edward Meador and Julia A. Pet- rov of the Office of International Education has been greatly appreciated. The advice and counsel of George \V. Bonham, Executive Director, and Robert Black, Director, of Programs, of the Council on Learning has also been helpful and consistent; and, at Educational Testing Serike the substantive and financial support provided by Winton H. Manning, Senior Vice President, Research and Development, has been critical. From the outset of the project in early 1979, we have been assisted by a distinguished Assessment Committee and wish to express our gratitude for the valuable guidance of these outstanding scholars: Robert F. Dern- , berger, Professor of Economics, University of Michigan; A. David Hill, Professsor of Geography, University of Colorado; William). McGuire, Professor cif Psychology, Yale University; William H. McNeill, Robert A. Milli- ken Distinguished Service Professor of History, University of Chicago; Richard C. Snyder, formerly Mershon Professor of Education and Public Policy and Director, Mershon Center, The Ohio State University; Judith V. Torney=Purta; Professor of Psychology, University of Illinois at Chicago Circle; Richard G. Tucker, Jr., Director, Center fin Applied Linguistics; Immanuel 7allerstein, Distinguished Professor of Sociology and Chairman, Department of Sociology, State University oi New York at Binghamton. The cooperation of staff and students at 18) colleges must also be recognized. Coordinators at each college drew samples of students, enlisted their participation, and administered the survey to them. These institutions ate listed in Appendix F. Finally, we must recognize unusual assistance from members of the staff at Educational Testing Service. Bruce Kaplan carried out our data analyses with exceptional speed, accuracy, and understanding, andboth Paul Holland and'flon Rubin provided analytic 'vice at critical points. Cor...e Pilla and her staff keytaped the sur- vey responses on an impossible schedule. Al.:.a Jackson, Nat Hartshorne, Elsa Rosenthal, and Sally Sharp as- sisted in the writing of our reports, and Christine Sansone and her text processing staff ably kept track of a seemingly endless series of revisions. The knowledge, cooperation, and expertise of these colleagues contributed immeasurably to the conduct of the survey and the preparation of this teport. and the authors extend their deepest,appreciation to each one. Thomas S. Barrows Stephen F. Klein John L. D. Clark 7 ii ,P4 Introduction Surveys usually seek to establish levels of some well-defined phenomenon in a popula- don of interest. Consumer-choice behavior, knowledge of American history, and even some political attitudes are fairly well-defined examples. Furthermore, methods for as- sessing them art, generally available and supported by considerable experience. Global understanding by contrast has not been defined, although it has been the subject of a considerable literature. Measures of global understanding either have not existed or have not been recognized because of this. Consequently, this report is not the usual survey fare. It covers the developnient of c:.:finitions of global understanding and the development and revision of survey mea- sures. It also describes analyses that examine the.fit between the original logical defini- tions and empirical, psychological models of the behavior defined. Because these steps were necessary, they are documented here, rendering the report unusually complex on the one hand but acceptably complete as a final report of the Global Understanding Project on the other. Part I of this report discusses the development of the survey measures. It introduces the difficulties encountered in defining the concept of global understanding and traces the refinement of the definition and resultant survey measures. Part II deals with the procedures used to conduct the survey. It focuses on the scientific methods used to iden- tify and select participating colleges and universities as well as the process of data collec- don. Part III presents a detailed analysis of the results and examines global understand- ing in terms of its components and their interrelationshipsknowledge, student back- ground, language learning, attitudes and perceptions. Part IV reviews and.summarizes the survey's findings. Readers who arc not interested in methodological techniques or complete, detailed survey results may prefer to refer to Hartshorne's synopsis. This emphasizes results and reduces the treatment of complex techniques that may nGt be of general interest. 'Barrows, Thomas S . Stephen F. Klein. and John L I). Clark with Nathaniel Hartshorne, What College Student, Know and New Rochelle, NY.. Change Magazine Press. 1981; E&WV Series V Believe Aboxt Thew World. iii Part I Survey Measures Although much has been written about global understanding and its numerous aliases, link has been done to define it. Indeed, using the pronoun "it" may constitute a-gross oversimplification. From the outset of This study, we have assumed that global understanding is more sensibly and tractibly approached as a multifaceted construct with components in both cognitive and affective domains. At the level of starting assumptions, it was clear that we would need to develop instru- ments to measure various domains of knowledge, skills, abilities, and so forth; and also attitudes, perceptions, interests, and other affective phenomena. In ..ddition to this intention of measuring many potential but as yet unspecified components of global understanding, the project's original objectives included exploration of some likely correlates of global understanding. Although correlates clearly could not be interpreted as causes, it was hoped that comprehensive background and experience data would provide some suggestions of how individual %Trianon in levels of global understanding comes about. Foreign language study and for- eign language proficiency were prime candidates here because of the relatively common belief that foreign language study or proficiency contributes to the understanding of other cultu:es. In the following sections we describe the efforts that were undertaken to provide instruments for the survey. The lengths of the sections reflect the difficulty of adequately covering an ill-defined phenomenonglobal understanding. Defining the cognitive components to be covered was pri- marily a rational, intellectual problem, while the affective area and foreign language proficiency presented primarily methodological challenges. The following sections, therefore, describe efforts that vary a good deal in terms of a rational or an analytic, empirical approach. a Chapter 1 Development of the Test of Knowledge Stephen F. Klein and Sheila M. Ager The. assignment to develop a test of global knowkdge immediately raised three questions: From what, or whose, perspective was the test to be developed? What knowlcdgc and skills did global undersianding,compiise? At what level of sophistication was global understanding to be tested? In order rte., insure that a broad range of perspectives was considered in answering these questions (and many others concerning the'test's development), the project staff decided to enlist the help of several scholars, who came to constitute an assessment committee. Yet the range of likely agsNcrs would have to be known first in order to decide what kind of committee to appoint. To the question of perspective, answers could plausibly be given that ranged from an.American perspective to a global one. In defense of the latter, one could argue that national, interest and eth: nocentrism were themselves major impediments to the achievement of global understanding and that the agenda for global understanding, therefore, had to be written by those who were not biased by these considerations. Conversely, one could argue that there is no such thing, in practice, as a universal outlook, that every individual has an outlook modifiad by, if not rooted in, experiences of time, place, social position, and so on, and that global undemanding, therefore, cannot bc inde- pendent of those who perceive it. Not only the content of global understanding would be at issue, but the relative importance of the various aspects of content as well. The range of answers to the question.of perspective ultimately left us with two choices regarding the appointment of an assessment committee. The first was to appoint committee members from around the world, on the assumption that the test they would develop would be the best practical approximation to a universal instrument. The second was to recruit committcc members froyn the United States whose world-mindedness would help to insure that a test relevant to American con: corns was not at the same time parochial. The second course of action was taken, since financial con- straints would have prevented all but a token effort to appoint committcc members from abroad . ; and since there seemed to be virtual unanimity within the global education community that Ameri- can college students knew too little about even American concerns, much less those of other nations. Te the questions of what knowlcdgc and what skills global understanding comprised, there was a better-defined range of answers to the former than the latter. Frott esen cursory reading in this area, if-Became apparent that there were two fundamentally different approaches to structuring the knowledge domain, although they intersected at numerous points. The first approach was based the established curricular traditions of international relations and area studie courses. The second regions. Partisans of was based on the concept of global issues that transcended particular nation the latter sometimes exaggerated the differences between issues on the one hand and nations (or in- ternational actors) and regions on the other. Clearly a knowlcdgc of international relations of one or more of the world's areas had to include knowledge of issues, even if those issues wcrc not consid- 'red "whole ", that is, in their full disciplinaryand geographical complexity. Just as clearly, know- ledge of issue5'.had .tolulucicitnowledge of the international arena-in which these -issues would or would not be worked out, as well as knowlcdgc of the areas of the world in wl.ich these issues im- pacted. A knowlcdgc of issues, therefore, could not possibly be devoid of references to particular in- stitutional structures and relations, or to partiCular times, places, and cultures, even if these were considered'to be somehow subsidiary to the issues themselves. And yet it would not be accurate to 2 10