Texts in Quantitative Political Analysis Series Editor: Justin Esarey Alessia Damonte Fedra Negri Editors Causality in Policy Studies A Pluralist Toolbox Texts in Quantitative Political Analysis Series Editor Justin Esarey, Dept of Politics, FM Kirby Hall 319 Wake Forest University Winston Salem, NC, USA This series covers the novel application of quantitative and mathematical methods to substantive problems in political science as well as the further extension, development, and adaptation of these methods to make them more useful for applied political science researchers. Books in this series make original contributions to political methodology and substantive political science, while serving as educational resources for independent practitioners and analysts working in the field. This series fills the needs of faculty, students, and independent practitioners as they develop and apply new quantitative research techniques or teach them to others. Books in this series are designed to be practical and easy-to-follow. Ideally, an independent reader should be able to replicate the authors’ analysis and follow any in-text examples without outside help. Some of the books will focus largely on instructing readers how to use software such as R or Stata. For textbooks, example data and (if appropriate) software code will be supplied by the authors for readers. This series welcomes proposals for monographs, edited volumes, textbooks, and professional titles. Alessia Damonte • Fedra Negri Editors Causality in Policy Studies A Pluralist Toolbox Editors Alessia Damonte Fedra Negri Social and Political Sciences University of Milan University of Milan Milano, Italy MILANO, Milano, Italy University of Milan- Bicocca Milano, Italy This book is an open access publication. ISSN 2730-9614 ISSN 2730-9622 (electronic) Texts in Quantitative Political Analysis ISBN 978-3-031-12981-0 ISBN 978-3-031-12982-7 (eBook) https://doi.org/10.1007/978-3-031-12982-7 © The Editor(s) (if applicable) and The Author(s) 2023 Open Access This book is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. The images or other third party material in this book are included in the book's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the book's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland Preface How can we think of causation in policy research? Almost any research tradition provides a different answer. For instance, emphasis can be placed either on the pro- cess leading to a policy outcome or on its underlying conditions. A process can be either observable or unobservable, and the underlying relevant conditions can be understood as single factors or complex configurations. Either samples, popula- tions, or single cases can be invoked as the proper empirical ground for grasping them. Evidence can be arranged to either claim relevance or irrelevance. These dif- ferences reflect as many distinct assumptions about the shape of causation and build as many research strategies. Causality in Policy Studies equips researchers to meet two related challenges in the field. First, algorithms for data analysis embed selected assumptions about causation that often remain unspoken. Knowing these assumptions is crucial to understanding how algorithms can be appropriately employed and eventually com- bined to compensate for their blind spots and weaknesses. Second, policy research is carried out within various disciplines (such as political science, sociology, eco- nomics, management, and administration), each often married to particular tradi- tions. The book addresses the technical drive of such differentiation. In doing so, it provides the opportunity for researchers of any stripe to familiarize themselves with the strategies on which other streams build their claims. In short, the book shows how to learn from different causal techniques, apply them consciously, and possibly make them speak to each other to get a better sense of findings. For this purpose, it structures the journey into causal knowledge in three stages. First, it introduces the foundational issues of causation (Chaps. 1 and 2). Then, it exposes the inner working of selected techniques for causal analysis (Chaps. 3, 4, 5, 6, 7, 8 and 9). Last, it considers some incompatibilities and complementari- ties among techniques to improve causal knowledge (Chaps. 10 and 11). The red thread connecting all chapters is a reasonable realist stance. All share the tenets that causation is factual and entails generative and transfer processes unfold- ing at different levels of reality. Moreover, the chapters agree that causation can be known. Hypothetical statements about its manifestations, direction, and conditions can be given a testable shape. They also agree that causal statements should be v vi Preface believed when logically and empirically compelling. The book’s commitment to methodological pluralism follows from these tenets. The complexity of causal phe- nomena is such that no single technique can grasp its entirety. Still, each technique can illuminate particular facets in response to a precise research question. Indeed, asking whether a factor can yield one outcome differs from asking how it happens or under which conditions it obtains, and each response calls for adequate analytic tools. When pieced together, these responses can offer a better account of the phe- nomena of interest. Methodological pluralism can deliver on the promise of better knowledge if the strengths and weaknesses of each technique are understood and tackled. To this end, each substantive chapter clarifies the research question a technique can answer, the research design and data treatment the technique requires for credible results, and the domain of validity of its findings. Wherever possible, a replicable example illus- trates the deployment of the analysis as the sequence of operations and actual deci- sions. Of course, this selection of techniques is far from exhaustive of the methodological variety of policy studies. Nevertheless, this suite provides sharp insight into the different strategies to establish the tenability of a causal statement. As such, it can offer guidance beyond the boundaries of this book. The edited format of the book aims at providing highly usable and solid knowl- edge for policy assessment and evaluation to MA students, PhD students, scholars, and practitioners in policy-related fields. Thus, each chapter is authored by a recog- nized scholar from different backgrounds, generations, and perspectives. Such a diverse yet “close-knit” team is essential to the volume. A single author could hardly have covered such a range of techniques with comparable expertise. Public policies are tools and governance systems to tackle collective problems. Good policies call for a generation of open-minded scholars and practitioners will- ing to understand and learn from research conducted in different fields and capable of handling the techniques in their toolbox consciously and carefully. We hope you will have a good time going through the chapters. Enjoy your journey! MILANO, Milano, Italy Alessia Damonte Fedra Negri Acknowledgments Every book is a collective enterprise and some more than others. Our first thanks go to the many students who have compelled this project and shaped it during our courses. They have been the real drivers of this effort, and we are more than grateful for how they kept our motivation high over the many months of writing and revis- ing. Heartfelt thanks also go to Licia Papavero, Francesco Zucchini, and the board of the Ph.D. in Political Studies of the University of Milan. Their constant support to the Summer School in “Research Strategies in Policy Studies” (ReSPoS) has proven vital to the maturation of this project, which consolidates an experience dat- ing back to 2013—now, a seemingly distant past. The School and the project, in turn, would not have been possible without the financial contributions of the Compagnia di San Paolo (Turin, Italy) through the Network for the Advancement of Social and Political Sciences (NASP) directed by Maurizio Ferrera. To them go our sincere gratitude. We also are greatly indebted to Springer’s Senior Editor for Economics, Political Science, and Public Administration, Lorraine Klimowich, and the Editor of the ‘Textbook on Political Analysis’ series, Justin Esarey. Their precious suggestions and faultless encouragement have been fundamental to final- izing a project intentionally positioned at the crossroad of many disciplines and research standards. Our further debt of gratitude is owed to Luigi Curini and the Standing Groups “MetRiSP—Research Methods for Political Science” and “Political Science & Public Policy” of the SISP—Italian Society of Political Science. We have treasured their feedbacks on earlier versions and their backing the main idea. Finally, we gratefully acknowledge the financial support of Protego—an advanced project funded by the European Research Council—Grant agreement n°694632. University of Milan Alessia Damonte Milan, Italy Fedra Negri vii Contents 1 Introduction: The Elephant of Causation and the Blind Sages . . . . . 1 Alessia Damonte and Fedra Negri 2 Causation in the Social Realm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Daniel Little 3 Counterfactuals with Experimental and Quasi-Experimental Variation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Erich Battistin and Marco Bertoni 4 Correlation Is Not Causation, Yet… Matching and Weighting for Better Counterfactuals . . . . . . . . . . . . . . . . . . . . . . 71 Fedra Negri 5 Getting the Most Out of Surveys: Multilevel Regression and Poststratification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Joseph T. Ornstein 6 Pathway Analysis, Causal Mediation, and the Identification of Causal Mechanisms . . . . . . . . . . . . . . . . . . . 123 Leonce Röth 7 Testing Joint Sufficiency Twice: Explanatory Qualitative Comparative Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Alessia Damonte 8 Causal Inference and Policy Evaluation from Case Studies Using Bayesian Process Tracing . . . . . . . . . . . . . . 187 Andrew Bennett 9 Exploring Interventions on Social Outcomes with In Silico, Agent-Based Experiments . . . . . . . . . . . . . . . . . . . . . . . 217 Flaminio Squazzoni and Federico Bianchi ix x Contents 10 The Many Threats from Mechanistic Heterogeneity That Can Spoil Multimethod Research . . . . . . . . . . . . . . . . . . . . . . . . 235 Markus B. Siewert and Derek Beach 11 Conclusions. Causality Between Plurality and Unity . . . . . . . . . . . . . 259 Alessia Damonte and Fedra Negri