THE CHANGING GOVERNANCE OF THE SCIENCES Sociology of the Sciences Yearbook VOLUME XXVI ManagingEditor: Peter Weingart, Universität Bielefeld, Germany EditorialBoard: Yaron Ezrahi, The Israel Democracy Institute, Jerusalem, Israel Ulrike Felt, Institute fu¨r Wissenschaftstheorie und Wissenschaftsforschung, Vienna, Austria Michael Hagner, Max-Planck-Institut fu¨r Wissenschaftsgeschichte, Berlin, Germany Stephen H. Hilgartner, Cornell University, Ithaca, U.S.A. Sheila Jasanoff, Harvard University, Cambridge, MA, U.S.A. Sabine Maasen, Wissenschaftsforschung/Wissenschaftssoziologie, Basel, Switzerland Everett Mendelsohn, Harvard University, Cambridge, MA, U.S.A. Helga Nowotny, ETH Zu¨rich, Zu¨rich, Switzerland Hans-Jo¨rg Rheinberger, Max-Planck Institut fu¨r Wissenschaftsgeschichte, Berlin, Germany Terry Shinn, GEMAS Maison des Sciences de l’Homme, Paris, France Richard D.Whitley, Manchester Business School, University of Manchester, United Kingdom Bjo¨rn Wittrock, SCASSS, Uppsala, Sweden Thetitlespublishedinthisseriesarelistedattheendofthisvolume. THE CHANGING GOVERNANCE OF THE SCIENCES The Advent of Research Evaluation Systems Edited by RICHARD WHITLEY UniversityofManchester,UK and JOCHEN GLÄSER University of Lancaster, UK LibraryofCongressControlNumber:2008920619 ISBN978-1-4020-6745-7(HB) ISBN978-1-4020-6746-4(e-book) PublishedbySpringer, P.O.Box17,3300AADordrecht,TheNetherlands. www.springer.com Printedonacid-freepaper AllRightsReserved © 2007Springer Science+Business Media B.V. Nopartofthisworkmaybereproduced,storedinaretrievalsystem,ortransmitted inanyformorbyanymeans,electronic,mechanical,photocopying,microfilming,recording orotherwise,withoutwrittenpermissionfromthePublisher,withtheexception ofanymaterialsuppliedspecificallyforthepurposeofbeingentered andexecutedonacomputersystem,forexclusiveusebythepurchaserofthework. PREFACE This book arose from a conversation between Lars Engwall and Richard Whitley during the 2003 European Group for Organization Studies Colloquium in Copenhagen about important topics for future research. We agreed then, and this has been amply confirmed by subsequent events, that the proliferation of research evalua- tion schemes, especially in Europe, was an important feature of the changing rela- tionships between the state, universities and scientific research more generally, which needed systematic and comparative analysis. With the support of the Bank of Sweden Tercentenary Foundation, this discussion led to workshops in Uppsala and Manchester involving colleagues from Australia, Germany and the Netherlands at which the main framework for such an analysis was developed. This framework then formed the basis for the conference at Bielefeld in 2005 at which earlier versions of most of the papers in this volume were presented and extensively discussed. We are very grateful for the support of the PRIME network of the European Commission and the Max Planck Institute for the History of Science for this con- ference, as well as the invaluable efforts of Peter Weingart of the University of Bielefeld. We are also greatly indebted to the members of the Editorial Board who commented on various drafts of these papers as well as to Aant Elzinga, Stefan Kuhlmann, Philippe Laredo, Arie Rip and Nic Vonortas who acted as referees. In preparing the volume for publication we have been greatly assisted by Kathryn Morrison and would also like to acknowledge the help of Kate Barker in liaising with the PRIME network, authors, and referees. Richard Whitley Jochen Gläser v TABLE OF CONTENTS List of Contributors.................................................................................................xiii Introduction Chapter 1 Changing Governance of the Public Sciences: The Consequences of Establishing Research Evaluation Systems for Knowledge Production in Different Countries and Scientific Fields......................3 Richard Whitley Introduction..........................................................................................3 Characteristics of Research Evaluation Systems and their Consequences...........................................................................6 The Consequences of Strong Research Evaluation Systems in Different Public Science Systems.......................................10 The Effects of Strong Research Evaluation Systems on Different Kinds of Scientific Fields...................................19 Conclusions........................................................................................24 References .........................................................................................25 Universities as Strategic Actors: State Delegation and Changing Patterns of Evaluation Chapter 2 The Visible Hand versus the Invisible Hand: The Allocation of Research Resources in Swedish Universities.................................31 Lars Engwall and Thorsten Nybom Introduction........................................................................................31 Institutional Control...........................................................................32 Entry Control..........................................................................32 Institutional Rules...................................................................34 Conclusions............................................................................37 Input Control......................................................................................37 Screening of Candidates.........................................................37 Allocation Procedures.............................................................38 Conclusions............................................................................42 Output Control...................................................................................43 Conclusions........................................................................................45 References .........................................................................................47 vii viii TABLE OF CONTENTS Chapter 3 Prospective and Retrospective Evaluation Systems in Context: Insights from Japan: .......................................................51 Robert Kneller Introduction........................................................................................51 University Research and Prospective Peer Review ...........................52 The University Research System............................................52 The Organisation of University Research and the Academic Reward Systems.........................................54 Research Funding and Prospective Peer Review....................55 The Advent of Retrospective Research Evaluation............................66 Final Remarks....................................................................................69 References..........................................................................................70 Appendix............................................................................................72 Chapter 4 Elite Through Rankings – The Emergence of the Enterprising University...........................................................................................75 Peter Weingart and Sabine Maasen Enterprising University Ahead?.........................................................75 Universities: Elements of Organisational Actorhood.........................77 Becoming an Entrepreneurial Actor: Elite Universities.....................80 Displaying the Entrepreneurial Actor: University Rankings..............83 Enterprising University in the Audit Society.....................................94 References..........................................................................................98 Chapter 5 The Social Construction of Bibliometric Evaluations......................101 Jochen Gläser and Grit Laudel A Constructivist Approach to Bibliometric Evaluation Tools.........101 The Modalities of Bibliometric Evaluation Methods.......................102 Marketing by Dropping Modalities..................................................105 The Demand for Modality-free Bibliometric Evaluations...............108 Working on Modalities....................................................................112 Amateur Bibliometrics.....................................................................116 Conclusions......................................................................................118 References........................................................................................120 Weak and Strong Research Evaluation Systems Chapter 6 Evaluation Without Evaluators: The Impact of Funding Formulae on Australian University Research..................................127 Jochen Gläser and Grit Laudel The Elusive Effects of Research Evaluation Systems......................127 Approach..........................................................................................129 TABLE OF CONTENTS ix Research Strategy – the Challenge of Causal Attribution.....129 Methodology and Methods – the Challenge of Empirical Identification.........................................132 Adaptations to a Strong Research Evaluation System.....................136 The Situation of Universities................................................136 Adaptation to the Research Funding Environment...............138 Consequences for Academics...............................................140 Adaptation by Academics.....................................................142 Changes in Knowledge Production.......................................145 Intended and Unintended Effects of Strong RES ............................146 References........................................................................................149 Chapter 7 The Basic State of Research in Germany: Conditions of Knowledge Production Pre-Evaluation .......................................153 Stefan Lange Introduction......................................................................................153 The Changing Governance of Academic Research in Germany .......154 University and Faculty Evaluation and Steering of Research..........157 Academic Responses to the Changing Context of Research............160 Time......................................................................................160 Funding.................................................................................161 Evaluations and Incentive Schemes......................................163 Profile-building.....................................................................164 Adaptation Patterns...............................................................165 Concluding Remarks........................................................................168 References........................................................................................169 Chapter 8 Research Evaluation as Organisational Development: The Work of the Academic Advisory Council in Lower Saxony (FRG) .........171 Christof Schiene and Uwe Schimank Introduction......................................................................................171 Procedure.........................................................................................173 Research Ideals................................................................................177 Effects..............................................................................................184 References........................................................................................188 Chapter 9 Interfering Governance and Emerging Centres of Control: University Research Evaluation in the Netherlands ........................191 Barend van der Meulen Introduction......................................................................................191 Episode 1: The Construction of an Evaluation.................................193 Episode 2: Construction of an Evaluation Object and Centres of Control..............................................................................195 x TABLE OF CONTENTS Episode 3: Standardisation, Comparability and Second Order Dynamics..............................................................................197 Conclusions......................................................................................200 References........................................................................................202 Appendix..........................................................................................203 Chapter 10 Research Evaluation in Transition: Individual versus Organisational Assessment in Spain................................................205 Laura Cruz-Castro and Luis Sanz-Menéndez Introduction......................................................................................205 The Academic System: Governance and Funding...........................207 Size and Governance............................................................207 Funding.................................................................................209 Institutionalisation Processes and Characteristics of the Spanish RES.........................................................................211 Institutional Arrangements of the Research Evaluation System (RES)............................................................212 The National Agency for Evaluation and Foresight.............212 The National Commission for the Evaluation of Research Activity......................................................................214 New Developments in the Spanish RES...............................215 Feedback Effects Between Evaluation and Research Systems........218 References........................................................................................221 Chapter 11 Death by Peer Review?: The Impact of Results-Oriented Management in U.S. Research ........................................................225 Susan E. Cozzens Analytic Frameworks.......................................................................226 The U.S. Public Research System....................................................229 Forms of Funding.............................................................................231 Types of Research Evaluation Systems...........................................232 Mission Agency Example.....................................................233 Basic Research Agency Changes .........................................235 Consequences...................................................................................236 The Benefits of Competition ................................................236 Interdisciplinarity..................................................................237 Short-termism ......................................................................238 Strategic Alignment .............................................................238 Open Windows ....................................................................239 Conclusions......................................................................................240 References........................................................................................241 TABLE OF CONTENTS xi Concluding Reflections Chapter 12 The Social Orders of Research Evaluation Systems........................245 Jochen Gläser A Sociological Approach to Research Evaluation...........................245 The Problem of Governing Science.................................................246 The Place of RES in the Governance by Social Orders...................249 Three Rationales for RES.....................................................249 Intrusive RES........................................................................252 Competitive RES..................................................................254 Responses by Universities....................................................256 The Interactions of Hierarchies and Competitions with Scientific Communities................................................258 Success and Possible Failures of RES..................................258 ‘Competition Failure’ of Quasi-market RES........................259 ‘Hierarchy Failure’ of Intrusive RES....................................262 Incentives for Future Research.........................................................263 References........................................................................................264 Author Index...........................................................................................................267