Deep Learning Ian Goodfellow Yoshua Bengio Aaron Courville Contents Website vii Acknowledgments viii Notation xi 1 Introduction 1 1.1 Who Should Read This Book? . . . . . . . . . . . . . . . . . . . . 8 1.2 Historical Trends in Deep Learning . . . . . . . . . . . . . . . . . 11 I Applied Math and Machine Learning Basics 29 2 Linear Algebra 31 2.1 Scalars, Vectors, Matrices and Tensors . . . . . . . . . . . . . . . 31 2.2 Multiplying Matrices and Vectors . . . . . . . . . . . . . . . . . . 34 2.3 Identity and Inverse Matrices . . . . . . . . . . . . . . . . . . . . 36 2.4 Linear Dependence and Span . . . . . . . . . . . . . . . . . . . . 37 2.5 Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.6 Special Kinds of Matrices and Vectors . . . . . . . . . . . . . . . 40 2.7 Eigendecomposition . . . . . . . . . . . . . . . . . . . . . . . . . . 42 2.8 Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . 44 2.9 The Moore-Penrose Pseudoinverse . . . . . . . . . . . . . . . . . . 45 2.10 The Trace Operator . . . . . . . . . . . . . . . . . . . . . . . . . 46 2.11 The Determinant . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 2.12 Example: Principal Components Analysis . . . . . . . . . . . . . 48 3 Probability and Information Theory 53 3.1 Why Probability? . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 i CONTENTS 3.2 Random Variables . . . . . . . . . . . . . . . . . . . . . . . . . . 56 3.3 Probability Distributions . . . . . . . . . . . . . . . . . . . . . . . 56 3.4 Marginal Probability . . . . . . . . . . . . . . . . . . . . . . . . . 58 3.5 Conditional Probability . . . . . . . . . . . . . . . . . . . . . . . 59 3.6 The Chain Rule of Conditional Probabilities . . . . . . . . . . . . 59 3.7 Independence and Conditional Independence . . . . . . . . . . . . 60 3.8 Expectation, Variance and Covariance . . . . . . . . . . . . . . . 60 3.9 Common Probability Distributions . . . . . . . . . . . . . . . . . 62 3.10 Useful Properties of Common Functions . . . . . . . . . . . . . . 67 3.11 Bayes’ Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 3.12 Technical Details of Continuous Variables . . . . . . . . . . . . . 71 3.13 Information Theory . . . . . . . . . . . . . . . . . . . . . . . . . . 72 3.14 Structured Probabilistic Models . . . . . . . . . . . . . . . . . . . 75 4 Numerical Computation 80 4.1 Overflow and Underflow . . . . . . . . . . . . . . . . . . . . . . . 80 4.2 Poor Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . 82 4.3 Gradient-Based Optimization . . . . . . . . . . . . . . . . . . . . 82 4.4 Constrained Optimization . . . . . . . . . . . . . . . . . . . . . . 93 4.5 Example: Linear Least Squares . . . . . . . . . . . . . . . . . . . 96 5 Machine Learning Basics 98 5.1 Learning Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . 99 5.2 Capacity, Overfitting and Underfitting . . . . . . . . . . . . . . . 110 5.3 Hyperparameters and Validation Sets . . . . . . . . . . . . . . . . 120 5.4 Estimators, Bias and Variance . . . . . . . . . . . . . . . . . . . . 122 5.5 Maximum Likelihood Estimation . . . . . . . . . . . . . . . . . . 131 5.6 Bayesian Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . 135 5.7 Supervised Learning Algorithms . . . . . . . . . . . . . . . . . . . 139 5.8 Unsupervised Learning Algorithms . . . . . . . . . . . . . . . . . 145 5.9 Stochastic Gradient Descent . . . . . . . . . . . . . . . . . . . . . 150 5.10 Building a Machine Learning Algorithm . . . . . . . . . . . . . . 152 5.11 Challenges Motivating Deep Learning . . . . . . . . . . . . . . . . 154 II Deep Networks: Modern Practices 165 6 Deep Feedforward Networks 167 6.1 Example: Learning XOR . . . . . . . . . . . . . . . . . . . . . . . 170 6.2 Gradient-Based Learning . . . . . . . . . . . . . . . . . . . . . . . 176 ii CONTENTS 6.3 Hidden Units . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 6.4 Architecture Design . . . . . . . . . . . . . . . . . . . . . . . . . . 196 6.5 Back-Propagation and Other Differentiation Algorithms . . . . . 203 6.6 Historical Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 7 Regularization for Deep Learning 228 7.1 Parameter Norm Penalties . . . . . . . . . . . . . . . . . . . . . . 230 7.2 Norm Penalties as Constrained Optimization . . . . . . . . . . . . 237 7.3 Regularization and Under-Constrained Problems . . . . . . . . . 239 7.4 Dataset Augmentation . . . . . . . . . . . . . . . . . . . . . . . . 240 7.5 Noise Robustness . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 7.6 Semi-Supervised Learning . . . . . . . . . . . . . . . . . . . . . . 243 7.7 Multi-Task Learning . . . . . . . . . . . . . . . . . . . . . . . . . 244 7.8 Early Stopping . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246 7.9 Parameter Tying and Parameter Sharing . . . . . . . . . . . . . . 253 7.10 Sparse Representations . . . . . . . . . . . . . . . . . . . . . . . . 254 7.11 Bagging and Other Ensemble Methods . . . . . . . . . . . . . . . 256 7.12 Dropout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 7.13 Adversarial Training . . . . . . . . . . . . . . . . . . . . . . . . . 268 7.14 Tangent Distance, Tangent Prop, and Manifold Tangent Classifier 270 8 Optimization for Training Deep Models 274 8.1 How Learning Differs from Pure Optimization . . . . . . . . . . . 275 8.2 Challenges in Neural Network Optimization . . . . . . . . . . . . 282 8.3 Basic Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 8.4 Parameter Initialization Strategies . . . . . . . . . . . . . . . . . 301 8.5 Algorithms with Adaptive Learning Rates . . . . . . . . . . . . . 306 8.6 Approximate Second-Order Methods . . . . . . . . . . . . . . . . 310 8.7 Optimization Strategies and Meta-Algorithms . . . . . . . . . . . 317 9 Convolutional Networks 330 9.1 The Convolution Operation . . . . . . . . . . . . . . . . . . . . . 331 9.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 9.3 Pooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339 9.4 Convolution and Pooling as an Infinitely Strong Prior . . . . . . . 345 9.5 Variants of the Basic Convolution Function . . . . . . . . . . . . 347 9.6 Structured Outputs . . . . . . . . . . . . . . . . . . . . . . . . . . 358 9.7 Data Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360 9.8 Efficient Convolution Algorithms . . . . . . . . . . . . . . . . . . 362 9.9 Random or Unsupervised Features . . . . . . . . . . . . . . . . . 363 iii CONTENTS 9.10 The Neuroscientific Basis for Convolutional Networks . . . . . . . 364 9.11 Convolutional Networks and the History of Deep Learning . . . . 371 10 Sequence Modeling: Recurrent and Recursive Nets 373 10.1 Unfolding Computational Graphs . . . . . . . . . . . . . . . . . . 375 10.2 Recurrent Neural Networks . . . . . . . . . . . . . . . . . . . . . 378 10.3 Bidirectional RNNs . . . . . . . . . . . . . . . . . . . . . . . . . . 395 10.4 Encoder-Decoder Sequence-to-Sequence Architectures . . . . . . . 396 10.5 Deep Recurrent Networks . . . . . . . . . . . . . . . . . . . . . . 398 10.6 Recursive Neural Networks . . . . . . . . . . . . . . . . . . . . . . 400 10.7 The Challenge of Long-Term Dependencies . . . . . . . . . . . . . 402 10.8 Echo State Networks . . . . . . . . . . . . . . . . . . . . . . . . . 405 10.9 Leaky Units and Other Strategies for Multiple Time Scales . . . . 408 10.10 The Long Short-Term Memory and Other Gated RNNs . . . . . . 410 10.11 Optimization for Long-Term Dependencies . . . . . . . . . . . . . 414 10.12 Explicit Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . 418 11 Practical Methodology 423 11.1 Performance Metrics . . . . . . . . . . . . . . . . . . . . . . . . . 424 11.2 Default Baseline Models . . . . . . . . . . . . . . . . . . . . . . . 427 11.3 Determining Whether to Gather More Data . . . . . . . . . . . . 428 11.4 Selecting Hyperparameters . . . . . . . . . . . . . . . . . . . . . . 429 11.5 Debugging Strategies . . . . . . . . . . . . . . . . . . . . . . . . . 438 11.6 Example: Multi-Digit Number Recognition . . . . . . . . . . . . . 442 12 Applications 445 12.1 Large Scale Deep Learning . . . . . . . . . . . . . . . . . . . . . . 445 12.2 Computer Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . 454 12.3 Speech Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . 460 12.4 Natural Language Processing . . . . . . . . . . . . . . . . . . . . 463 12.5 Other Applications . . . . . . . . . . . . . . . . . . . . . . . . . . 479 III Deep Learning Research 488 13 Linear Factor Models 491 13.1 Probabilistic PCA and Factor Analysis . . . . . . . . . . . . . . . 492 13.2 Independent Component Analysis (ICA) . . . . . . . . . . . . . . 493 13.3 Slow Feature Analysis . . . . . . . . . . . . . . . . . . . . . . . . 495 13.4 Sparse Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498 iv CONTENTS 13.5 Manifold Interpretation of PCA . . . . . . . . . . . . . . . . . . . 501 14 Autoencoders 504 14.1 Undercomplete Autoencoders . . . . . . . . . . . . . . . . . . . . 505 14.2 Regularized Autoencoders . . . . . . . . . . . . . . . . . . . . . . 506 14.3 Representational Power, Layer Size and Depth . . . . . . . . . . . 510 14.4 Stochastic Encoders and Decoders . . . . . . . . . . . . . . . . . . 511 14.5 Denoising Autoencoders . . . . . . . . . . . . . . . . . . . . . . . 512 14.6 Learning Manifolds with Autoencoders . . . . . . . . . . . . . . . 517 14.7 Contractive Autoencoders . . . . . . . . . . . . . . . . . . . . . . 523 14.8 Predictive Sparse Decomposition . . . . . . . . . . . . . . . . . . 525 14.9 Applications of Autoencoders . . . . . . . . . . . . . . . . . . . . 526 15 Representation Learning 528 15.1 Greedy Layer-Wise Unsupervised Pretraining . . . . . . . . . . . 530 15.2 Transfer Learning and Domain Adaptation . . . . . . . . . . . . . 538 15.3 Semi-Supervised Disentangling of Causal Factors . . . . . . . . . 543 15.4 Distributed Representation . . . . . . . . . . . . . . . . . . . . . . 548 15.5 Exponential Gains from Depth . . . . . . . . . . . . . . . . . . . 555 15.6 Providing Clues to Discover Underlying Causes . . . . . . . . . . 556 16 Structured Probabilistic Models for Deep Learning 560 16.1 The Challenge of Unstructured Modeling . . . . . . . . . . . . . . 561 16.2 Using Graphs to Describe Model Structure . . . . . . . . . . . . . 565 16.3 Sampling from Graphical Models . . . . . . . . . . . . . . . . . . 582 16.4 Advantages of Structured Modeling . . . . . . . . . . . . . . . . . 584 16.5 Learning about Dependencies . . . . . . . . . . . . . . . . . . . . 584 16.6 Inference and Approximate Inference . . . . . . . . . . . . . . . . 585 16.7 The Deep Learning Approach to Structured Probabilistic Models 586 17 Monte Carlo Methods 592 17.1 Sampling and Monte Carlo Methods . . . . . . . . . . . . . . . . 592 17.2 Importance Sampling . . . . . . . . . . . . . . . . . . . . . . . . . 594 17.3 Markov Chain Monte Carlo Methods . . . . . . . . . . . . . . . . 597 17.4 Gibbs Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 601 17.5 The Challenge of Mixing between Separated Modes . . . . . . . . 601 18 Confronting the Partition Function 607 18.1 The Log-Likelihood Gradient . . . . . . . . . . . . . . . . . . . . 608 18.2 Stochastic Maximum Likelihood and Contrastive Divergence . . . 609 v CONTENTS 18.3 Pseudolikelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . 617 18.4 Score Matching and Ratio Matching . . . . . . . . . . . . . . . . 619 18.5 Denoising Score Matching . . . . . . . . . . . . . . . . . . . . . . 621 18.6 Noise-Contrastive Estimation . . . . . . . . . . . . . . . . . . . . 622 18.7 Estimating the Partition Function . . . . . . . . . . . . . . . . . . 625 19 Approximate Inference 633 19.1 Inference as Optimization . . . . . . . . . . . . . . . . . . . . . . 635 19.2 Expectation Maximization . . . . . . . . . . . . . . . . . . . . . . 636 19.3 MAP Inference and Sparse Coding . . . . . . . . . . . . . . . . . 637 19.4 Variational Inference and Learning . . . . . . . . . . . . . . . . . 640 19.5 Learned Approximate Inference . . . . . . . . . . . . . . . . . . . 653 20 Deep Generative Models 656 20.1 Boltzmann Machines . . . . . . . . . . . . . . . . . . . . . . . . . 656 20.2 Restricted Boltzmann Machines . . . . . . . . . . . . . . . . . . . 658 20.3 Deep Belief Networks . . . . . . . . . . . . . . . . . . . . . . . . . 662 20.4 Deep Boltzmann Machines . . . . . . . . . . . . . . . . . . . . . . 665 20.5 Boltzmann Machines for Real-Valued Data . . . . . . . . . . . . . 678 20.6 Convolutional Boltzmann Machines . . . . . . . . . . . . . . . . . 685 20.7 Boltzmann Machines for Structured or Sequential Outputs . . . . 687 20.8 Other Boltzmann Machines . . . . . . . . . . . . . . . . . . . . . 688 20.9 Back-Propagation through Random Operations . . . . . . . . . . 689 20.10 Directed Generative Nets . . . . . . . . . . . . . . . . . . . . . . . 694 20.11 Drawing Samples from Autoencoders . . . . . . . . . . . . . . . . 712 20.12 Generative Stochastic Networks . . . . . . . . . . . . . . . . . . . 716 20.13 Other Generation Schemes . . . . . . . . . . . . . . . . . . . . . . 717 20.14 Evaluating Generative Models . . . . . . . . . . . . . . . . . . . . 719 20.15 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 721 Bibliography 723 Index 780 vi Website www.deeplearningbook.org This book is accompanied by the above website. The website provides a variety of supplementary material, including exercises, lecture slides, corrections of mistakes, and other resources that should be useful to both readers and instructors. vii Acknowledgments This book would not have been possible without the contributions of many people. We would like to thank those who commented on our proposal for the book and helped plan its contents and organization: Guillaume Alain, Kyunghyun Cho, Çağlar Gülçehre, David Krueger, Hugo Larochelle, Razvan Pascanu and Thomas Rohée. We would like to thank the people who offered feedback on the content of the book itself. Some offered feedback on many chapters: Martín Abadi, Guillaume Alain, Ion Androutsopoulos, Fred Bertsch, Olexa Bilaniuk, Ufuk Can Biçici, Matko Bošnjak, John Boersma, Greg Brockman, Alexandre de Brébisson, Pierre Luc Carrier, Sarath Chandar, Pawel Chilinski, Mark Daoust, Oleg Dashevskii, Laurent Dinh, Stephan Dreseitl, Jim Fan, Miao Fan, Meire Fortunato, Frédéric Francis, Nando de Freitas, Çağlar Gülçehre, Jurgen Van Gael, Javier Alonso García, Jonathan Hunt, Gopi Jeyaram, Chingiz Kabytayev, Lukasz Kaiser, Varun Kanade, Akiel Khan, John King, Diederik P. Kingma, Yann LeCun, Rudolf Mathey, Matías Mattamala, Abhinav Maurya, Kevin Murphy, Oleg Mürk, Roman Novak, Augustus Q. Odena, Simon Pavlik, Karl Pichotta, Kari Pulli, Roussel Rahman, Tapani Raiko, Anurag Ranjan, Johannes Roith, Mihaela Rosca, Halis Sak, César Salgado, Grigory Sapunov, Yoshinori Sasaki, Mike Schuster, Julian Serban, Nir Shabat, Ken Shirriff, Andre Simpelo, Scott Stanley, David Sussillo, Ilya Sutskever, Carles Gelada Sáez, Graham Taylor, Valentin Tolmer, An Tran, Shubhendu Trivedi, Alexey Umnov, Vincent Vanhoucke, Marco Visentini-Scarzanella, David Warde-Farley, Dustin Webb, Kelvin Xu, Wei Xue, Ke Yang, Li Yao, Zygmunt Zając and Ozan Çağlayan. We would also like to thank those who provided us with useful feedback on individual chapters: Notation: Zhang Yuanhang. • Chapter 1, Introduction: Yusuf Akgul, Sebastien Bratieres, Samira Ebrahimi, • Charlie Gorichanaz, Brendan Loudermilk, Eric Morris, Cosmin Pârvulescu viii CONTENTS and Alfredo Solano. Chapter 2, Linear Algebra: Amjad Almahairi, Nikola Banić, Kevin Bennett, • Philippe Castonguay, Oscar Chang, Eric Fosler-Lussier, Andrey Khalyavin, Sergey Oreshkov, István Petrás, Dennis Prangle, Thomas Rohée, Colby Toland, Massimiliano Tomassoli, Alessandro Vitale and Bob Welland. Chapter 3, Probability and Information Theory: John Philip Anderson, Kai • Arulkumaran, Vincent Dumoulin, Rui Fa, Stephan Gouws, Artem Oboturov, Antti Rasmus, Alexey Surkov and Volker Tresp. Chapter 4, Numerical Computation: Tran Lam An, Ian Fischer, and Hu • Yuhuang. Chapter 5, Machine Learning Basics: Dzmitry Bahdanau, Nikhil Garg, • Makoto Otsuka, Bob Pepin, Philip Popien, Emmanuel Rayner, Kee-Bong Song, Zheng Sun and Andy Wu. Chapter 6, Deep Feedforward Networks: Uriel Berdugo, Fabrizio Bottarel, • Elizabeth Burl, Ishan Durugkar, Jeff Hlywa, Jong Wook Kim, David Krueger and Aditya Kumar Praharaj. Chapter 7, Regularization for Deep Learning: Kshitij Lauria, Inkyu Lee, • Sunil Mohan and Joshua Salisbury. Chapter 8, Optimization for Training Deep Models: Marcel Ackermann, • Rowel Atienza, Andrew Brock, Tegan Maharaj, James Martens, Klaus Strobl and Martin Vita. Chapter 9, Convolutional Networks: Martín Arjovsky, Eugene Brevdo, Kon- • stantin Divilov, Eric Jensen, Asifullah Khan, Mehdi Mirza, Alex Paino, Eddie Pierce, Marjorie Sayer, Ryan Stout and Wentao Wu. Chapter 10, Sequence Modeling: Recurrent and Recursive Nets: Gökçen • Eraslan, Steven Hickson, Razvan Pascanu, Lorenzo von Ritter, Rui Rodrigues, Dmitriy Serdyuk, Dongyu Shi and Kaiyu Yang. Chapter 11, Practical Methodology: Daniel Beckstein. • Chapter 12, Applications: George Dahl and Ribana Roscher. • Chapter 15, Representation Learning: Kunal Ghosh. • ix