BIG DATA MINING AND COMPLEXITY THE SAGE QUANTITATIVE RESEARCH KIT Big Data Mining and Complexity by Brian Castellani and Rajeev Rajaram is the 11th volume in The SAGE Quantitative Research Kit. This book can be used together with the other titles in the Kit as a comprehensive guide to the process of doing quantitative research, but is equally valuable on its own as a practical introduction to Data Mining and ‘Big Data’. Editors of The SAGE Quantitative Research Kit: Malcolm Williams – Cardiff University, UK Richard D. Wiggins – UCL Social Research Institute, UK D. Betsy McCoach – University of Connecticut, USA Founding editor: The late W. Paul Vogt – Illinois State University, USA BIG DATA MINING AND COMPLEXITY BRIAN C. CASTELLANI RAJEEV RAJARAM THE SAGE QUANTITATIVE RESEARCH KIT SAGE Publications Ltd  Brian C. Castellani and Rajeev Rajaram 2021 1 Oliver’s Yard 55 City Road This volume published as part of The SAGE Quantitative London EC1Y 1SP Research Kit (2021), edited by Malcolm Williams, Richard D. Wiggins and D. Betsy McCoach. SAGE Publications Inc. 2455 Teller Road Apart from any fair dealing for the purposes of research, Thousand Oaks, California 91320 private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act, 1988, this publication SAGE Publications India Pvt Ltd may not be reproduced, stored or transmitted in any form, B 1/I 1 Mohan Cooperative Industrial Area or by any means, without the prior permission in writing of Mathura Road the publisher, or in the case of reprographic reproduction, in accordance with the terms of licences issued by the Copyright Licensing Agency. Enquiries concerning New Delhi 110 044 reproduction outside those terms should be sent to SAGE Publications Asia-Pacific Pte Ltd the publisher. 3 Church Street #10-04 Samsung Hub Singapore 049483 Library of Congress Control Number: 2020943076 Editor: Jai Seaman Assistant editor: Charlotte Bush British Library Cataloguing in Publication data Production editor: Manmeet Kaur Tura Copyeditor: QuADS Prepress Pvt Ltd A catalogue record for this book is available from the Proofreader: Elaine Leek British Library Indexer: Cathryn Pritchard Marketing manager: Susheel Gokarakonda Cover design: Shaun Mercier Typeset by: C&M Digitals (P) Ltd, Chennai, India Printed in the UK ISBN 978-1-5264-2381-8 At SAGE we take sustainability seriously. Most of our products are printed in the UK using responsibly sourced papers and boards. When we print overseas, we ensure sustainable papers are used as measured by the PREPS grading system. We undertake an annual audit to monitor our sustainability. Dedicated to Maggie, Ruby and Swathi Contents List of Figures xiii About the Authors xv 1 Introduction 1 The Joys of Travel 2 Data Mining and Big Data Travel 2 Part I: Thinking Critically and Complex 3 Organisation of Part I 4 Part II: The Tools and Techniques of Data Mining 5 SAGE Quantitative Research Kit 7 COMPLEX-IT and the SACS Toolkit 7 The Airline Industry: A Case Study 8 PART I THINKING CRITICALLY AND COMPLEX 11 2 The Failure of Quantitative Social Science 13 Quantitative Social Science, Then and Now 15 The Three Phases of Science 15 What You Should Have Learned in Statistics Class 16 So, Why Didn’t You Learn These Things? 19 Changing the Social Life of Data Analysis 21 3 What Is Big Data? 23 Big Data as Information Society 24 Big Data as Global Network Society 25 The Socio-Cybernetic Web of Big Data 26 Big Data Databases 28 The Failed Promise of Big Data 31 4 What Is Data Mining? 33 A Bit of Data Mining History 34 The Data Mining Process 35 viii BIG DATA MINING AND COMPLEXITY The ‘Black Box’ of Data Mining 36 Validity and Reliability 38 The Limits of Normalised Probability Distributions 39 Fitting Models to Data 40 Data Mining’s Various Tasks 40 5 The Complexity Turn 43 Mapping the Complexity Turn 44 Data Mining and Big Data as Complexity Science 46 Top Ten List About Complexity 46 Number 1 47 Number 2 47 Number 3 47 Number 4 47 Number 5 48 Number 6 49 Number 7 49 Number 8 50 Number 9 50 Number 10 50 PART II THE TOOLS AND TECHNIQUES OF DATA MINING 53 6 Case-Based Complexity: A Data Mining Vocabulary 55 Case-Based Complexity 56 COMPLEX-IT and the SACS Toolkit 57 The Ontology of Big Data 59 The Archaeology of Big Data Ontologies 59 The Formalisms of Case-Based Complexity 63 What Is a Case? 63 Two Definitions of a Vector 64 Cataloguing and Grouping Case Profiles 65 Mathematical Distance Between Cases 66 Cataloguing Case Profiles 67 Diversity of Case Profiles 69 The Notion of Time t 69 Profiles That Vary With Time 69 Static Clustering in Time 70 Dynamic Clustering of Trajectories 70 CONTENTS ix A Vector Field 71 The State Space 71 A Discrete Vector Field of Velocities of Trajectories 72 Why Do We Need a Vector Field? 75 7 Classification and Clustering 79 Top Ten Airlines 80 Classification Versus Clustering 80 Classification Schemes 81 Decision Tree Induction 82 Nearest Neighbour Classifier 83 Artificial Neural Networks 83 Support Vector Machines 84 Clustering 84 Hierarchical Clustering Methods 85 Partitioning Methods 86 Probability Density-Based Methods 87 Soft Computing Methods 87 8 Machine Learning 91 The Smart Airline Industry 92 What Is Machine Intelligence? 93 Machine Intelligence for Big Data 93 Examples of Data Mining Methods Based on Machine Intelligence 94 Artificial Neural Networks 95 Genetic Algorithms and Evolutionary Computation 100 Swarming Travel Routes 100 Overview of Swarm Intelligence 101 9 Predictive Analytics and Data Forecasting 105 Predictive Analytics and Data Forecasting: A Very Brief History 106 Overview of Techniques 107 Bayesian Statistics 107 Bayesian Parameter Estimation 108 Bayesian Hypothesis Testing 110 Decision Trees 112 Neural Networks 112 Regression (Linear, Non-Linear and Logistic) 112