Machine Learning in Python for Process Systems Engineering Achieve operational excellence using process data Ankur Kumar, Jesus Flores-Cerrillo 1 ` Machine Learning in Python for Process Systems Engineering Achieving operational excellence using process data Ankur Kumar Jesus Flores-Cerrillo ` Dedicated to our spouses, family, friends, motherland, and all the data-science enthusiasts ` Machine Learning in Python for Process Systems Engineering www.MLforPSE.com Copyright © 2022 Ankur Kumar All rights reserved. No part of this book may be reproduced or transmitted in any form or in any manner without the prior written permission of the authors. . Every effort has been made in the preparation of this book to ensure the accuracy of the information presented and obtain permissions for usage of copyrighted materials. However, the authors make no warranties, expressed or implied, regarding errors or omissions and assume no legal liability or responsibility for loss or damage resulting from the use of information contained in this book . To request permissions, contact the authors at [email protected] First published: February 2022 ` About the Authors Ankur Kumar holds a PhD degree (2016) in Process Systems Engineering from the University of Texas at Austin and a bachelor’s degree (2012) in Chemical Engineering from the Indian Institute of Technology Bombay. He currently works at Linde in the Advanced Digital Technologies & Systems Group in Linde’s Center of Excellence, where he has developed several in-house machine learning-based monitoring and process control solutions for Linde’s hydrogen and air-separation plants. Ankur’s tools have won several awards both within and outside Linde. Most recently, one of his tools, PlantWatch (a plantwide fault detection and diagnosis tool), received the 2021 Industry 4.0 Award by the Confederation of Industry of the Czech Republic. Ankur has authored or co-authored more than 10 peer-reviewed journal papers (in the areas of data-driven process modeling and optimization), is a frequent reviewer for many top-ranked Journals, and has served as Session Chair at several international conferences. Ankur also served as an Associate Editor of the Journal of Process Control from 2019 to 2021. Jesus Flores-Cerrillo is currently an Associate Director - R&D at Linde and manages the Advanced Digital Technologies & Systems Group in Linde’s Center of Excellence. He has over 20 years of experience in the development and implementation of monitoring technologies and advanced process control & optimization solutions. Jesus holds a PhD degree in Chemical Engineering from McMaster University and has authored or co-authored more than 40 peer-reviewed journal papers in the areas of multivariate statistics and advanced process control among others. His team develops and implements novel plant monitoring, machine learning, IIOT solutions to improve the efficiency and reliability of Linde’s processes. Jesus’s team received the Artificial Intelligence and Advanced Analytics Leadership 2020 award from the National Association of Manufacturers’ Manufacturing Leadership Council. ` Note to the readers Jupyter notebooks and Spyder scripts with complete code implementations are available for download at https://github.com/ML-PSE/Machine_Learning_for_PSE. Code updates when necessary, will be made and updated on the GitHub repository. Updates to the book’s text material will be available free of cost on Leanpub (www.leanpub.com). We would greatly appreciate any information about any corrections and/or typos in the book. A minor pending issue In mathematical expressions, bold-face letters refer to vectors and matrices and regular- faced letters refer to scalar variables. However, this distinction does not appear correctly in some of the expressions. These expressions will be corrected soon, and readers would be notified of the updates accordingly. Nonetheless, this minor pending issue would not prevent the readers from correctly understanding the material. ` Table of Contents • Preface • Part 1 Introduction and Fundamentals • Chapter 1 Machine Learning for Process Systems Engineering 1 o 1.1 What are Process Systems ▪ 1.1.1 Characteristics of process data o 1.2 What is Machine Learning ▪ 1.2.1 Machine learning workflow ▪ 1.2.2 Type of machine learning systems o 1.3 Machine Learning Applications in Process Industry ▪ 1.3.1 Decision hierarchy levels in a process plant ▪ 1.3.2 Application areas o 1.4 ML Solution Deployment o 1.5 The Future of Process Data Science • Chapter 2 The Scripting Environment 20 o 2.1 Introduction to Python o 2.2 Introduction to Spyder and Jupyter o 2.3 Python Language: Basics o 2.4 Scientific Computing Packages: Basics ▪ 2.4.1 Numpy ▪ 2.4.2 Pandas o 2.5 Typical ML Script • Chapter 3 Machine Learning Model Development: Workflow and Best Practices 39 o 3.1 ML Model Development Workflow o 3.2 Data Pre-processing: Data Transformation ▪ 3.2.1 (Robust) Data centering & scaling ▪ 3.2.2 Feature extraction ▪ 3.2.3 Feature engineering ▪ 3.2.4 Workflow automation via pipelines o 3.3 Model Evaluation ▪ 3.3.1 Regression metrics ▪ 3.3.2 Classification metrics ▪ 3.3.3 Holdout method / cross-validation ▪ 3.3.4 Residual analysis o 3.4 Model Tuning ▪ 3.4.1 Overfitting & underfitting ▪ 3.4.2 Train/validation/test split ▪ 3.3.3 K-fold cross-validation ▪ 3.4.4 Regularization ▪ 3.4.5 Hyperparameter optimization via GridSearchCV • Chapter 4 Data Pre-processing: Cleaning Process Data 67 o 4.1 Signal De-noising ▪ 4.1.1 Moving window average filter ▪ 4.1.2 SG filter ` o 4.2 Variable Selection/Feature Selection ▪ 4.2.1 Filter methods ▪ 4.2.2 Wrapper methods ▪ 4.2.3 Embedded methods o 4.3 Outlier Handling ▪ 4.3.1 Univariate methods ▪ 4.3.2 Multivariate methods ▪ 4.3.3 Data-mining methods o 4.4 Handling Missing Data • Part 2 Classical Machine Learning Methods • Chapter 5 Dimension Reduction and Latent Variable Methods (Part 1) 90 o 5.1 PCA: An Introduction ▪ 5.1.1 Mathematical background ▪ 5.1.2 Dimensionality reduction for polymer manufacturing process o 5.2 Process Monitoring via PCA for Polymer Manufacturing Process ▪ 5.2.1 Process monitoring/fault detection indices ▪ 5.2.2 Fault detection ▪ 5.2.3 Fault diagnosis o 5.3 Variants of Classical PCA ▪ 5.3.1 Dynamic PCA ▪ 5.3.2 Multiway PCA ▪ 5.3.3 Kernel PCA o 5.4 PLS: An Introduction ▪ 5.4.1 Mathematical background o 5.5 Soft Sensing via PLS for Pulp & Paper Manufacturing Process o 5.6 Process monitoring via PLS for Polyethylene Manufacturing Process ▪ 5.6.1 Fault detection indices ▪ 5.6.2 Fault detection o 5.7 Variants of Classical PLS • Chapter 6 Dimension Reduction and Latent Variable Methods (Part 2) 120 o 6.1 ICA: An Introduction ▪ 6.1.1 Mathematical background ▪ 6.1.2 Complex chemical process: Tennessee Eastman Process ▪ 6.1.3 Deciding number of ICs o 6.2 Process Monitoring via ICA for Tennessee Eastman Process ▪ 6.2.1 Fault detection indices ▪ 6.2.2 Fault detection o 6.3 FDA: An Introduction ▪ 6.3.1 Mathematical background ▪ 6.3.2 Dimensionality reduction for Tennessee Eastman Process o 6.4 Fault Classification via FDA for Tennessee Eastman Process • Chapter 7 Support Vector Machines & Kernel-based Learning 142 o 7.1 SVMs: An Introduction ▪ 7.1.1 Mathematical background ▪ 7.1.2 Hard margin vs soft margin classification o 7.2 The Kernel Trick for Nonlinear Data ▪ 7.2.1 Mathematical background o 7.3 SVDD: An Introduction ` ▪ 7.3.1 Mathematical background ▪ 7.3.2 OC-SVM vs SVDD ▪ 7.3.3 Bandwidth parameter and SVDD illustration o 7.4 Process Fault Detection via SVDD o 7.5 SVR: An Introduction ▪ 7.5.1 Mathematical background o 7.6 Soft Sensing via SVR in a Polymer Processing Plant o 7.7 Soft Sensing via SVR for Debutanizer Column in a Petroleum Refinery • Chapter 8 Finding Groups in Process Data: Clustering & Mixture Modeling 170 o 8.1 Clustering: An Introduction ▪ 8.1.1 Multimode semiconductor manufacturing process o 8.2 Centroid-based Clustering: K-Means ▪ 8.2.1 Determining the number of clusters via elbow method ▪ 8.2.2 Silhouette analysis for quantifying clusters quality ▪ 8.2.3 Pros and cons o 8.3 Density-based Clustering: DBSCAN ▪ 8.3.3 Pros and cons o 8.4 Probabilistic Clustering: Gaussian Mixtures ▪ 8.4.1 Mathematical background ▪ 8.4.2 Determining the number of clusters o 8.5 Multimode Process Monitoring via GMM for Semiconductor Manufacturing Process ▪ 8.5.1 Fault detection indices ▪ 8.5.2 Fault detection • Chapter 9 Decision Trees & Ensemble Learning 192 o 9.1 Decision Trees: An Introduction ▪ 9.1.1 Mathematical background o 9.2 Random Forests: An Introduction ▪ 9.2.1 Mathematical background o 9.3 Soft Sensing via Random Forest in Concrete Construction Industry ▪ 9.3.1 Feature importances o 9.4 Introduction to Ensemble Learning ▪ 9.4.1 Bagging ▪ 9.4.2 Boosting o 9.5 Effluent Quality Prediction in Wastewater Treatment Plant via XGBoost • Chapter 10 Other Useful Classical ML Techniques 214 o 10.1 KDE: An Introduction ▪ 10.1.1 Mathematical background ▪ 10.1.2 Deciding KDE hyperparameters o 10.2 Determining Monitoring Metric Control Limit via KDE o 10.3 kNN: An Introduction ▪ 10.3.1 Mathematical background ▪ 10.3.2 Deciding kNN hyperparameters ▪ 10.3.3 Applications of kNN for process systems o 10.4 Process Fault Detection via kNN for semiconductor Manufacturing Process o 10.5 Combining ML Techniques ` • Part 3 Artificial Neural Networks & Deep Learning • Chapter 11 Feedforward Neural Networks 230 o 11.1 ANN: An Introduction ▪ 11.1.1 Deep learning ▪ 11.1.2 TensorFlow o 11.2 Process Modeling via FFNN for Combined Cycle Power Plant o 11.3 Mathematical Background ▪ 11.3.1 Activation functions ▪ 11.3.2 Loss functions & cost functions ▪ 11.3.3 Gradient descent optimization ▪ 11.3.4 Epochs & batch-size ▪ 11.3.5 Backpropagation ▪ 11,3,6 Vanishing/Exploding gradients o 11.4 Nonlinearity in Neural Nets (Width vs Depth) o 11.5 Neural Net Hyperparameter Optimization o 11.6 Strategies for Improved Network Training ▪ 11.6.1 Early stopping ▪ 11.6.2 Regularization ▪ 11.6.3 Initialization ▪ 11.6.4 Batch normalization o 11.7 Soft Sensing via FFNN for Debutanizer Column in a Petroleum Refinery o FFNN Modeling Guidelines • Chapter 12 Recurrent Neural Networks 256 o 12.1 RNN: An Introduction ▪ 12.1.1 RNN outputs ▪ 12.1.2 LSTM networks o 12.2 System Identification via LSTM RNN for SISO Heater System o 12.3 Mathematical Background o 12.4 Stacked/Deep RNNs o 12.5 Fault Classification vis LSTM for Tennessee Eastman Process o 12.6 Predictive Maintenance using LSTM Networks ▪ 12.6.1 Failure prediction using LSTM ▪ 12.6.2 Remaining useful life (RUL) prediction using LSTM • Chapter 13 Reinforcement Learning 285 o 13.1 Reinforcement Learning: An Introduction ▪ 13.1.1 RL for process control o 13.2 RL Terminology & Mathematical Concepts ▪ 13.2.1 Environment and Markov decision process ▪ 13.2.2 Reward and return ▪ 13.2.3 Policy ▪ 13.2.4 Value function ▪ 13.2.5 Bellman equation o 13.3 Fundamentals of Q-learning o 13.4 Deep RL & Actor-Critic Framework ▪ 13.4.1 Deep Q-learning ▪ 13.4.2 Policy gradient methods ▪ 13.4.3 Actor-Critic framework o 13.5 Deep Deterministic Policy Gradient (DDPG) ▪ 13.5.1 Replay memory