Industrial Applications of Machine Learning Chapman & Hall/CRC Data Mining and Knowledge Series Series Editor: Vipin Kumar Data Classification Algorithms and Applications Charu C. Aggarwal Healthcare Data Analytics Chandan K. Reddy and Charu C. Aggarwal Accelerating Discovery Mining Unstructured Information for Hypothesis Generation Scott Spangler Event Mining Algorithms and Applications Tao Li Text Mining and Visualization Case Studies Using Open-Source Tools Markus Hofmann and Andrew Chisholm Graph-Based Social Media Analysis Ioannis Pitas Data Mining A Tutorial-Based Primer, Second Edition Richard J. Roiger Data Mining with R Learning with Case Studies, Second Edition Luís Torgo Social Networks with Rich Edge Semantics Quan Zheng and David Skillicorn Large-Scale Machine Learning in the Earth Sciences Ashok N. Srivastava, Ramakrishna Nemani, and Karsten Steinhaeuser Data Science and Analytics with Python Jesus Rogel-Salazar Feature Engineering for Machine Learning and Data Analytics Guozhu Dong and Huan Liu Exploratory Data Analysis Using R Ronald K. Pearson Human Capital Systems, Analytics, and Data Mining Robert C. Hughes Industrial Applications of Machine Learning Pedro Larrañaga et al For more information about this series please visit: https://www.crcpress.com/Chapman--HallCRC-Data- Mining-and-Knowledge-Discovery-Series/book-series/ CHDAMINODIS Industrial Applications of Machine Learning Pedro Larrañaga David Atienza Javier Diaz-Rozo Alberto Ogbechie Carlos Puerto-Santana Concha Bielza CRC Press Taylor & Francis Group 6000 Broken Sound Parkway NW, Suite 300 Boca Raton, FL 33487-2742 © 2019 by Taylor & Francis Group, LLC CRC Press is an imprint of Taylor & Francis Group, an Informa business No claim to original U.S. Government works Printed on acid-free paper Version Date: 20181003 International Standard Book Number-13: 978-0-8153-5622-6 (Hardback) This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint. Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers. For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com and the CRC Press Web site at http://www.crcpress.com Contents Preface xi 1 The Fourth Industrial Revolution 1 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.1 Industrie 4.0 . . . . . . . . . . . . . . . . . . . . . . . 5 1.1.2 Industrial Internet of Things . . . . . . . . . . . . . . 6 1.1.3 Other International Strategies. . . . . . . . . . . . . . . 7 1.2 Industry Smartization . . . . . . . . . . . . . . . . . . . . . . 9 1.2.1 At the Component Level. . . . . . . . . . . . . . . . . 9 1.2.2 At the Machine Level . . . . . . . . . . . . . . . . . . 10 1.2.3 At the Production Level . . . . . . . . . . . . . . . . . . 11 1.2.4 At the Distribution Level . . . . . . . . . . . . . . . . 12 1.3 Machine Learning Challenges and Opportunities within Smart Industries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.3.1 Impact on Business. . . . . . . . . . . . . . . . . . . . 13 1.3.2 Impact on Technology . . . . . . . . . . . . . . . . . . 15 1.3.3 Impact on People . . . . . . . . . . . . . . . . . . . . . 15 1.4 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . 16 2 Machine Learning 19 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.2 Basic Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.2.1 Descriptive Statistics . . . . . . . . . . . . . . . . . . . 23 2.2.1.1 Visualization and Summary of Univariate Data 24 2.2.1.2 Visualization and Summary of Bivariate Data 26 2.2.1.3 Visualization and Summary of Multivariate Data . . . . . . . . . . . . . . . . . . . . . . 26 2.2.1.4 Imputation of Missing Data. . . . . . . . . . 29 2.2.1.5 Variable Transformation . . . . . . . . . . . . 31 2.2.2 Inference . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.2.2.1 Parameter Point Estimation . . . . . . . . . 32 2.2.2.2 Parameter Confidence Estimation . . . . . . 36 2.2.2.3 Hypothesis Testing . . . . . . . . . . . . . . 36 2.3 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 2.3.1 Hierarchical Clustering. . . . . . . . . . . . . . . . . . 40 2.3.2 K-Means Algorithm . . . . . . . . . . . . . . . . . . . 42 v vi Contents 2.3.3 Spectral Clustering . . . . . . . . . . . . . . . . . . . . 43 2.3.4 Affinity Propagation . . . . . . . . . . . . . . . . . . . 45 2.3.5 Probabilistic Clustering . . . . . . . . . . . . . . . . . 46 2.4 Supervised Classification . . . . . . . . . . . . . . . . . . . . 49 2.4.1 Model Performance Evaluation . . . . . . . . . . . . . . 51 2.4.1.1 Performance Evaluation Measures . . . . . . . 51 2.4.1.2 Honest Performance Estimation Methods . . 56 2.4.2 Feature Subset Selection . . . . . . . . . . . . . . . . . 59 2.4.3 k-Nearest Neighbors . . . . . . . . . . . . . . . . . . . 65 2.4.4 Classification Trees . . . . . . . . . . . . . . . . . . . . . 67 2.4.5 Rule Induction . . . . . . . . . . . . . . . . . . . . . . 69 2.4.6 Artificial Neural Networks . . . . . . . . . . . . . . . . 72 2.4.7 Support Vector Machines . . . . . . . . . . . . . . . . 76 2.4.8 Logistic Regression . . . . . . . . . . . . . . . . . . . . 80 2.4.9 Bayesian Network Classifiers . . . . . . . . . . . . . . 82 2.4.9.1 Discrete Bayesian Network Classifiers . . . . 82 2.4.9.2 Continuous Bayesian Network Classifiers . . 89 2.4.10 Metaclassifiers . . . . . . . . . . . . . . . . . . . . . . 90 2.5 Bayesian Networks . . . . . . . . . . . . . . . . . . . . . . . . 94 2.5.1 Fundamentals of Bayesian Networks . . . . . . . . . . 94 2.5.2 Inference in Bayesian Networks . . . . . . . . . . . . . 100 2.5.2.1 Types of Inference . . . . . . . . . . . . . . . 100 2.5.2.2 Exact Inference . . . . . . . . . . . . . . . . 102 2.5.2.3 Approximate Inference . . . . . . . . . . . . . 107 2.5.3 Learning Bayesian Networks from Data . . . . . . . . 108 2.5.3.1 Learning Bayesian Network Parameters . . . 108 2.5.3.2 Learning Bayesian Network Structures. . . . . 111 2.6 Modeling Dynamic Scenarios with Bayesian Networks . . . . 115 2.6.1 Data Streams . . . . . . . . . . . . . . . . . . . . . . . 115 2.6.2 Dynamic, Temporal and Continuous Time Bayesian Networks . . . . . . . . . . . . . . . . . . . . . . . . . 119 2.6.3 Hidden Markov Models . . . . . . . . . . . . . . . . . 123 2.6.3.1 EvaluationoftheLikelihoodofanObservation Sequence . . . . . . . . . . . . . . . . . . . . 125 2.6.3.2 Decoding . . . . . . . . . . . . . . . . . . . . 126 2.6.3.3 Hidden Markov Model Training . . . . . . . . 127 2.7 Machine Learning Tools . . . . . . . . . . . . . . . . . . . . . 128 2.8 The Frontiers of Machine Learning . . . . . . . . . . . . . . . . 131 3 Applications of Machine Learning in Industrial Sectors 133 3.1 Energy Sector . . . . . . . . . . . . . . . . . . . . . . . . . . 133 3.1.1 Oil . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 3.1.2 Gas . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 3.2 Basic Materials Sector . . . . . . . . . . . . . . . . . . . . . . 136 3.2.1 Chemicals . . . . . . . . . . . . . . . . . . . . . . . . . 136 Contents vii 3.2.2 Basic Resources. . . . . . . . . . . . . . . . . . . . . . 138 3.3 Industrials Sector . . . . . . . . . . . . . . . . . . . . . . . . 139 3.3.1 Construction and Materials . . . . . . . . . . . . . . . . 141 3.3.2 Industrial Goods and Services . . . . . . . . . . . . . . . 141 3.4 Consumer Services Sector . . . . . . . . . . . . . . . . . . . . 143 3.4.1 Retail . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 3.4.2 Media . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 3.4.3 Tourism . . . . . . . . . . . . . . . . . . . . . . . . . . 144 3.5 Healthcare Sector . . . . . . . . . . . . . . . . . . . . . . . . 145 3.5.1 Cancer . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 3.5.2 Neuroscience . . . . . . . . . . . . . . . . . . . . . . . 148 3.5.3 Cardiovascular . . . . . . . . . . . . . . . . . . . . . . 149 3.5.4 Diabetes . . . . . . . . . . . . . . . . . . . . . . . . . . 150 3.5.5 Obesity . . . . . . . . . . . . . . . . . . . . . . . . . . 150 3.5.6 Bioinformatics . . . . . . . . . . . . . . . . . . . . . . 150 3.6 Consumer Goods Sector . . . . . . . . . . . . . . . . . . . . . . 151 3.6.1 Automobiles . . . . . . . . . . . . . . . . . . . . . . . . 151 3.6.2 Food and Beverages . . . . . . . . . . . . . . . . . . . 152 3.6.3 Personal and Household Goods . . . . . . . . . . . . . 155 3.7 Telecommunications Sector . . . . . . . . . . . . . . . . . . . 156 3.7.1 Software for Network Analysis . . . . . . . . . . . . . . 157 3.7.2 Data Transmission . . . . . . . . . . . . . . . . . . . . . 157 3.8 Utilities Sector . . . . . . . . . . . . . . . . . . . . . . . . . . 159 3.8.1 Utilities Generation . . . . . . . . . . . . . . . . . . . 159 3.8.2 Utilities Distribution . . . . . . . . . . . . . . . . . . . 160 3.9 Financial Services Sector . . . . . . . . . . . . . . . . . . . . . 161 3.9.1 Customer-Focused Applications . . . . . . . . . . . . . . 161 3.9.2 Operations-Focused Applications . . . . . . . . . . . . 162 3.9.3 Trading and Portfolio Management Applications . . . 163 3.9.4 Regulatory Compliance and Supervision Applications 163 3.10 Information Technology Sector . . . . . . . . . . . . . . . . . 164 3.10.1 Hardware and semi-conductors . . . . . . . . . . . . . 164 3.10.2 Software . . . . . . . . . . . . . . . . . . . . . . . . . . 165 3.10.3 Data Center Management . . . . . . . . . . . . . . . . 165 3.10.4 Cybersecurity . . . . . . . . . . . . . . . . . . . . . . . 166 4 Component-Level Case Study: Remaining Useful Life of Bearings 167 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 4.2 Ball Bearing Prognostics . . . . . . . . . . . . . . . . . . . . 168 4.2.1 Data-Driven Techniques . . . . . . . . . . . . . . . . . 168 4.2.2 PRONOSTIA Testbed . . . . . . . . . . . . . . . . . . 170 4.3 Feature Extraction from Vibration Signals . . . . . . . . . . 170 4.4 Hidden Markov Model-Based RUL Estimation . . . . . . . . 175 4.4.1 Hidden Markov Model Construction . . . . . . . . . . . 177 viii Contents 4.5 Results and Discussion . . . . . . . . . . . . . . . . . . . . . 179 4.5.1 RUL Results . . . . . . . . . . . . . . . . . . . . . . . 179 4.5.2 Interpretation of the Degradation Model . . . . . . . . 180 4.6 Conclusions and Future Research . . . . . . . . . . . . . . . . 181 4.6.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 181 4.6.2 Future Research . . . . . . . . . . . . . . . . . . . . . . 181 5 Machine-Level Case Study: Fingerprint of Industrial Motors 185 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 5.2 Performance of Industrial Motors as a Fingerprint . . . . . . 186 5.2.1 Improving Reliability Models with Fingerprints . . . . 186 5.2.2 Industrial Internet Consortium Testbed . . . . . . . . . 187 5.2.3 Testbed Dataset Description . . . . . . . . . . . . . . 193 5.3 Clustering Algorithms for Fingerprint Development . . . . . 194 5.3.1 Agglomerative Hierarchical Clustering . . . . . . . . . 195 5.3.2 K-means Clustering . . . . . . . . . . . . . . . . . . . 195 5.3.3 Spectral Clustering . . . . . . . . . . . . . . . . . . . . 196 5.3.4 Affinity Propagation . . . . . . . . . . . . . . . . . . . 196 5.3.5 Gaussian Mixture Model Clustering . . . . . . . . . . . 197 5.3.6 Implementation Details . . . . . . . . . . . . . . . . . 198 5.4 Results and Discussion . . . . . . . . . . . . . . . . . . . . . 198 5.5 Conclusions and Future Research . . . . . . . . . . . . . . . 205 5.5.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . 205 5.5.2 Future Research . . . . . . . . . . . . . . . . . . . . . 205 6 Production-Level Case Study: Automated Visual Inspection of a Laser Process 207 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 6.2 Laser Surface Heat Treatment . . . . . . . . . . . . . . . . . 210 6.2.1 Image Acquisition . . . . . . . . . . . . . . . . . . . . . 211 6.2.2 Response Time Requirement . . . . . . . . . . . . . . 215 6.3 Anomaly Detection-Based AVI System . . . . . . . . . . . . 215 6.3.1 Anomaly Detection Algorithms in Image Processing . 216 6.3.1.1 Probabilistic Anomaly Detection . . . . . . . . 217 6.3.1.2 Distance-Based Anomaly Detection . . . . . . 217 6.3.1.3 Reconstruction-Based Anomaly Detection . . 218 6.3.1.4 Domain-Based Anomaly Detection . . . . . . 219 6.3.2 Proposed Methodology. . . . . . . . . . . . . . . . . . 219 6.3.2.1 Feature Extraction . . . . . . . . . . . . . . . 222 6.3.2.2 Dynamic Bayesian Networks Implementation 225 6.3.2.3 Performance Assessment . . . . . . . . . . . . 227 6.4 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . 227 6.4.1 Performance of the AVI System . . . . . . . . . . . . . . 227 6.4.2 Interpretation of the Normality Model . . . . . . . . . 229 Contents ix 6.4.2.1 RelationshipsintheDynamicBayesianNetwork Structure . . . . . . . . . . . . . . . . . . . . 229 6.4.2.2 RelationshipsintheDynamicBayesianNetwork Parameters . . . . . . . . . . . . . . . . . . . 239 6.5 Conclusions and Future Research . . . . . . . . . . . . . . . 246 6.5.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . 246 6.5.2 Future Research . . . . . . . . . . . . . . . . . . . . . . 247 7 Distribution-Level Case Study: Forecasting of Air Freight Delays 249 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 249 7.2 Air Freight Process . . . . . . . . . . . . . . . . . . . . . . . . 251 7.2.1 Data Preprocessing . . . . . . . . . . . . . . . . . . . . 252 7.2.1.1 Simplification of Planned/Actual Times . . . 255 7.2.1.2 Transport Leg Reordering . . . . . . . . . . . . 257 7.2.1.3 Airport Simplification . . . . . . . . . . . . . 258 7.2.1.4 Normalizing the Length of Each Business Process . . . . . . . . . . . . . . . . . . . . . . 261 7.3 Supervised Classification Algorithms for Forecasting Delays . 262 7.3.1 k-Nearest Neighbors . . . . . . . . . . . . . . . . . . . 262 7.3.2 Classification Trees . . . . . . . . . . . . . . . . . . . . 263 7.3.3 Rule Induction . . . . . . . . . . . . . . . . . . . . . . 264 7.3.4 Artificial Neural Networks . . . . . . . . . . . . . . . . 265 7.3.5 Support Vector Machines . . . . . . . . . . . . . . . . 266 7.3.6 Logistic Regression . . . . . . . . . . . . . . . . . . . . . 267 7.3.7 Bayesian Network Classifiers . . . . . . . . . . . . . . . 267 7.3.8 Metaclassifiers . . . . . . . . . . . . . . . . . . . . . . 268 7.3.9 Implementation Details of Classification Algorithms . 270 7.4 Results and Discussion . . . . . . . . . . . . . . . . . . . . . 270 7.4.1 Compared Classifiers . . . . . . . . . . . . . . . . . . . . 271 7.4.2 Quantitative Comparison of Classifiers . . . . . . . . . 273 7.4.2.1 Multiple Hypothesis Testing . . . . . . . . . 274 7.4.2.2 Online Classification of Business Processes . 275 7.4.3 Qualitative Comparison of Classifiers . . . . . . . . . . . 277 7.4.3.1 C4.5 . . . . . . . . . . . . . . . . . . . . . . . . 277 7.4.3.2 RIPPER . . . . . . . . . . . . . . . . . . . . 284 7.4.3.3 Bayesian Network Classifiers . . . . . . . . . 284 7.4.4 Feature Subset Selection . . . . . . . . . . . . . . . . . 288 7.5 Conclusions and Future Research . . . . . . . . . . . . . . . 289 7.5.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . 289 7.5.2 Future Research . . . . . . . . . . . . . . . . . . . . . . 291 Bibliography 293 Index 325