APPLIED SOCIAL SCIENCE METHODOLOGY: AN INTRODUCTORY GUIDE John Gerring Dino Christenson Department of Political Science Boston University Forthcoming, Cambridge University Press, 2017 Draft: June 7, 2016 Word count (excluding Part IV): 140,000 Comments welcome! 1 Contents Preface ........................................................................................................................................................... 18 I. BUILDING BLOCKS ................................................................................................................................. 22 1. A Unified Framework ......................................................................................................................... 24 2. Arguments ............................................................................................................................................ 37 3. Concepts and Measures ...................................................................................................................... 58 4. Analyses ................................................................................................................................................ 78 II. CAUSALITY .............................................................................................................................................. 94 5. Causal Frameworks ............................................................................................................................. 95 6. Causal Hypotheses and Analyses ................................................................................................... 122 7. Experimental Designs ...................................................................................................................... 141 8. Large-N Observational Designs ..................................................................................................... 163 9. Case Study Designs .......................................................................................................................... 186 10. Diverse Tools of Causal Inference ............................................................................................ 202 III. PROCESS AND PRESENTATION ................................................................................................. 214 11. Reading and Reviewing ............................................................................................................... 215 12. Brainstorming ............................................................................................................................... 232 13. Data Gathering ............................................................................................................................. 245 14. Writing ........................................................................................................................................... 283 15. Speaking ......................................................................................................................................... 303 16. Ethics ............................................................................................................................................. 314 IV. STATISTICS (separate doc) ..................................................................................................................... 323 References ...................................................................................................................................................... 324 Index ............................................................................................................................................................... 335 2 Detailed Contents Notation ................................................................................................................................................... 16 Acknowledgments .................................................................................................................................. 17 Preface ........................................................................................................................................................... 18 A Wide-Ranging Approach ................................................................................................................... 19 Outline and Features .............................................................................................................................. 20 I. BUILDING BLOCKS ................................................................................................................................. 22 1. A Unified Framework ......................................................................................................................... 24 The Purpose of Unity ............................................................................................................................. 26 Examples .................................................................................................................................................. 28 Worker-training programs .............................................................................................................. 28 Social capital ................................................................................................................................... 29 Democracy ...................................................................................................................................... 30 Conclusions ..................................................................................................................................... 32 Key Terms................................................................................................................................................ 34 Inquiries .................................................................................................................................................... 34 Resources ................................................................................................................................................. 35 Web sites ......................................................................................................................................... 35 History and philosophy of social science ........................................................................................ 35 Worker-training programs .............................................................................................................. 35 Democracy ...................................................................................................................................... 36 Social capital ................................................................................................................................... 36 2. Arguments ............................................................................................................................................ 37 Descriptive Arguments .......................................................................................................................... 37 Figure 2.1: Taxonomy of Descriptive Arguments ....................................................................... 38 Associations .................................................................................................................................... 39 Figure 2.2: Possible Relationships among Two Factors ............................................................. 40 Syntheses ........................................................................................................................................ 41 Simple Typologies ........................................................................................................................... 42 Periodization ................................................................................................................................... 43 Matrix Typologies ............................................................................................................................ 43 Table 2.1: A Matrix Typology: Regime-types ............................................................................. 44 3 Taxonomies ..................................................................................................................................... 44 Table 2.2: Regime Taxonomy in Tabular Format ....................................................................... 45 Figure 2.3: Regime Taxonomy in Tree-Diagram Format ............................................................ 46 Causal Arguments ................................................................................................................................... 47 Other Arguments .................................................................................................................................... 48 Table 2.4: Typology of Arguments ............................................................................................. 49 Good Arguments .................................................................................................................................... 49 Table 2.5: Theorizing: General Criteria ...................................................................................... 50 Precision .......................................................................................................................................... 50 Generality ........................................................................................................................................ 50 Boundedness ................................................................................................................................... 51 Parsimony ....................................................................................................................................... 51 Logical Coherence ........................................................................................................................... 52 Commensurability ........................................................................................................................... 52 Innovation ....................................................................................................................................... 52 Relevance ........................................................................................................................................ 53 Conclusions ............................................................................................................................................. 53 Key Terms................................................................................................................................................ 55 Inquiries .................................................................................................................................................... 55 Resources ................................................................................................................................................. 56 Description, generally considered .................................................................................................. 56 Typologies, classification ................................................................................................................. 56 Causal arguments ............................................................................................................................ 57 3. Concepts and Measures ...................................................................................................................... 58 Concepts................................................................................................................................................... 58 Table 3.1: Criteria of Concept Formation ................................................................................... 59 Resonance ....................................................................................................................................... 59 Internal coherence .......................................................................................................................... 59 External Differentiation .................................................................................................................. 60 Theoretical utility ............................................................................................................................ 61 Consistency ..................................................................................................................................... 61 Strategies of Concept Formation.......................................................................................................... 61 4 Table 3.2: Democracy: Fundamental Attributes ....................................................................... 63 Minimal ........................................................................................................................................... 64 Maximal........................................................................................................................................... 65 Coda ................................................................................................................................................ 65 Measures ................................................................................................................................................... 65 Table 3.3: Measurement Strategies ........................................................................................... 66 Levels of abstraction ....................................................................................................................... 66 Scales ............................................................................................................................................... 67 Table 3.4: Typology of Scales ..................................................................................................... 68 Table 3.5: A Single Index with Multiple Interpretations ............................................................ 69 Aggregation ..................................................................................................................................... 69 Objectives........................................................................................................................................ 71 Conclusions ............................................................................................................................................. 72 Key Terms................................................................................................................................................ 73 Inquiries .................................................................................................................................................... 73 Resources ................................................................................................................................................. 76 Concepts.......................................................................................................................................... 76 Measures ......................................................................................................................................... 77 4. Analyses ................................................................................................................................................ 78 Definitions ............................................................................................................................................... 78 Figure 4.1: Time-Series Cross-Section Dataset .......................................................................... 80 Precision & Validity ................................................................................................................................ 81 Figure 4.2: Precision and Validity ............................................................................................... 83 Internal & External Validity .................................................................................................................. 85 Sample Representativeness .................................................................................................................... 85 Sample Size (N) ....................................................................................................................................... 86 Probability Sampling............................................................................................................................... 87 Non-probability Sampling ..................................................................................................................... 88 Missing-ness ............................................................................................................................................. 89 Conclusions ............................................................................................................................................. 90 Key Terms................................................................................................................................................ 92 Inquiries .................................................................................................................................................... 92 Resources ................................................................................................................................................. 93 5 Sampling .......................................................................................................................................... 93 II. CAUSALITY .............................................................................................................................................. 94 5. Causal Frameworks ............................................................................................................................. 95 Table 5.1: Causal Frameworks ................................................................................................... 96 Motivational Frameworks ...................................................................................................................... 96 Interests .......................................................................................................................................... 96 Norms .............................................................................................................................................. 97 Psychology ...................................................................................................................................... 98 Structural Frameworks ........................................................................................................................... 99 Material factors ............................................................................................................................... 99 Human capital/Demography ......................................................................................................... 100 Institutions .................................................................................................................................... 100 Interactive Frameworks ...................................................................................................................... 101 Adaptation .................................................................................................................................... 101 Coordination ................................................................................................................................. 102 Figure 5.1: Prisoner’s Dilemma ................................................................................................ 103 Diffusion ........................................................................................................................................ 103 Networks ....................................................................................................................................... 104 Figure 5.2: A Facebook Social Network .................................................................................... 105 Path dependence .......................................................................................................................... 106 Building a Theory................................................................................................................................. 107 An Example: Birth .......................................................................................................................... 107 Formal models .............................................................................................................................. 108 Figure 5.3: A Spatial Model of Vote Choice and Party Positioning .......................................... 110 Conclusions .......................................................................................................................................... 111 Key Terms............................................................................................................................................. 113 Inquiries ................................................................................................................................................. 113 Resources .............................................................................................................................................. 114 Causal frameworks, generally considered .................................................................................... 114 Interests (that which benefits an actor) ....................................................................................... 115 Norms (values, beliefs, ideology, culture, identity) ...................................................................... 115 Information (ideas, knowledge, learning) ..................................................................................... 116 6 Psychology (cognition, emotion, personality, genetics) ............................................................... 116 Material factors (resources, modernization, geography, technology) ......................................... 117 Institutions (formal rules and organizations) ................................................................................ 117 Human capital/Demography (health, education, skills, migration, life-cycle events) .................. 118 Diffusion (demonstration effects, exposure) ................................................................................ 118 Adaptation (feedback, competition, selection, evolution, efficiency, functionalism) .................. 119 Coordination (collective action, externalities, transaction costs) ................................................ 119 Networks (associational structures, social capital) ....................................................................... 120 Path-dependence (contingent moments leading to fixed trajectories) ....................................... 120 Simulations .................................................................................................................................... 120 Formal models .............................................................................................................................. 121 6. Causal Hypotheses and Analyses ................................................................................................... 122 Causality ................................................................................................................................................ 122 Figure 6.1: A Simple Causal Graph ........................................................................................... 125 Causal Graphs................................................................................................................................ 126 Criteria of a Causal Hypothesis.......................................................................................................... 126 Table 6.2: Causal Hypotheses: Criteria .................................................................................... 127 Clarity ............................................................................................................................................ 128 Manipulability ............................................................................................................................... 128 Precedence.................................................................................................................................... 129 Impact ........................................................................................................................................... 130 Mechanism .................................................................................................................................... 131 Causal Analysis ..................................................................................................................................... 131 Table 6.3: Causal Research Designs: Criteria ........................................................................... 132 Covariation .................................................................................................................................... 132 Figure 6.2: Covariational Patterns ............................................................................................ 134 Comparability ................................................................................................................................ 135 Conclusions .......................................................................................................................................... 136 Key Terms............................................................................................................................................. 137 Inquiries ................................................................................................................................................. 138 Resources .............................................................................................................................................. 140 7. Experimental Designs ...................................................................................................................... 141 7 Experiments With and Without Confounding ................................................................................ 141 Figure 7.1: Experimental Data without Confounders: An Illustration ..................................... 142 Figure 7.2: Experimental Data without Confounders: A Causal Graph .................................... 143 Figure 7.3: Experimental Data without Confounders: X/Y Plot ............................................... 144 Non-compliance ............................................................................................................................ 145 Figure 7.4: Experimental Data with Non-Compliance: An Illustration ..................................... 145 Contamination .............................................................................................................................. 146 Figure 7.5: Experimental Data with Contamination: An Illustration ........................................ 147 Compound treatments .................................................................................................................. 148 Figure 7.6: Causal Graph of Compound Treatment Confounding ........................................... 149 Varieties of Experiments .................................................................................................................... 149 Typology of designs ....................................................................................................................... 150 Table 7.1: A Typology of Experimental Designs ....................................................................... 152 Randomization mechanisms ......................................................................................................... 152 Research settings .......................................................................................................................... 152 Examples ............................................................................................................................................... 153 Employment discrimination .......................................................................................................... 153 Corruption control ........................................................................................................................ 154 Historic election campaigns .......................................................................................................... 155 Gender and leadership .................................................................................................................. 155 Conclusions .......................................................................................................................................... 157 Key Terms............................................................................................................................................. 159 Inquiries ................................................................................................................................................. 159 Resources .............................................................................................................................................. 161 Published work .............................................................................................................................. 161 Web sites ....................................................................................................................................... 162 8. Large-N Observational Designs ..................................................................................................... 163 Table 8.1: Large-N Observational Research Designs ................................................................ 164 Cross-sectional Designs ...................................................................................................................... 164 Table 8.2: Cross-sectional Design ............................................................................................. 164 Figure 8.1: Cross-sectional Data: A Typical Scenario ............................................................... 165 Figure 8.2: Causal Graph with Common-Cause Confounder ................................................... 166 8 Figure 8.3: Conditioning on a Common-Cause Confounder: An Illustration ........................... 167 Figure 8.4: Causal Graph with Mechanismic (post-treatment) Confounder ........................... 168 Figure 8.5: Causal Graph with Collider Confounder................................................................. 169 Figure 8.6: Causal Graph with Circular Confounding ............................................................... 170 Example ......................................................................................................................................... 170 Time-Series Designs ............................................................................................................................ 171 Table 8.3: Time-series Research Designs ................................................................................. 172 Figure 8.7: Time-series Data: A Typical Scenario ..................................................................... 172 Example ......................................................................................................................................... 174 Time-series Cross-section (TSCS) Designs ...................................................................................... 175 Example ......................................................................................................................................... 176 Table 8.4: Time-series Cross-section (TSCS) Designs ............................................................... 178 Regression-discontinuity (RD) designs ............................................................................................. 178 Examples ....................................................................................................................................... 179 Table 8.5: Regression-Discontinuity (RD) Design ..................................................................... 180 Instrumental-variable (IV) designs .................................................................................................... 180 Example ......................................................................................................................................... 181 Figure 8.8: Instrumental Variable (IV) Design .......................................................................... 182 Conclusions .......................................................................................................................................... 183 Key Terms............................................................................................................................................. 184 Inquiries ................................................................................................................................................. 184 Resources .............................................................................................................................................. 185 Regression-discontinuity (RD) designs .......................................................................................... 185 Instrumental variable (IV) designs ................................................................................................ 185 Observational designs, generally considered ............................................................................... 185 9. Case Study Designs .......................................................................................................................... 186 Case Selection ....................................................................................................................................... 187 Table 9.1: Case Selection Strategies ........................................................................................ 188 Exploratory .................................................................................................................................... 188 Diagnostic ...................................................................................................................................... 190 Omnibus Criteria ................................................................................................................................. 192 Intrinsic importance ...................................................................................................................... 192 Data availability ............................................................................................................................. 193 9 Logistics ......................................................................................................................................... 193 Case independence ....................................................................................................................... 193 Sample representativeness ........................................................................................................... 193 Within-case Analysis ............................................................................................................................ 194 Identifying a hypothesis/theory .................................................................................................... 194 Testing theories............................................................................................................................. 194 Temporality ................................................................................................................................... 196 Examining Background Assumptions ............................................................................................ 197 Strengths and weaknesses ................................................................................................................... 197 Table 9.2: Case Study and Large-N Research Designs .............................................................. 198 Key Terms............................................................................................................................................. 200 Inquiries ................................................................................................................................................. 200 Resources .............................................................................................................................................. 200 10. Diverse Tools of Causal Inference ............................................................................................ 202 Identifying and avoiding confounders .............................................................................................. 202 Figure 10.1: Typology of Confounders using Causal Graphs .................................................... 204 Research designs .................................................................................................................................. 205 Figure 10.2: Taxonomy of Causal Research Designs ................................................................ 205 And the Winner Is… ........................................................................................................................... 207 Multiple Methods ................................................................................................................................. 209 Conclusions .......................................................................................................................................... 211 Key Terms............................................................................................................................................. 212 Inquiries ................................................................................................................................................. 212 Resources .............................................................................................................................................. 212 Natural experiments ..................................................................................................................... 212 Multimethod Research ................................................................................................................. 212 III. PROCESS AND PRESENTATION ................................................................................................. 214 11. Reading and Reviewing ............................................................................................................... 215 Identifying Social Science Sources .................................................................................................... 215 Journals ......................................................................................................................................... 216 Books ............................................................................................................................................. 216 Working papers and blogs ............................................................................................................ 217 General observations .................................................................................................................... 217 10