Convolutional Neural Networks Deep Learning Lecture 4 Samuel Cheng School of ECE University of Oklahoma Spring, 2017 S. Cheng (OU-Tulsa) Convolutional Neural Networks Jan 2017 1 / 198 Table of Contents 1 Review 2 Babysitting your learning job 3 Overview and history of CNN 4 CNN basic 5 Case study 6 Some CNN tricks 7 Conclusions S. Cheng (OU-Tulsa) Convolutional Neural Networks Jan 2017 2 / 198 Presentation order S. Cheng (OU-Tulsa) Convolutional Neural Networks Jan 2017 3 / 198 Logistics HW1 is due today 5% per day penalty (of HW1) starting tomorrow Naim is the winner for the first HW with 3% overall bonus As extra “bonus” to the winner, I would like him to present his solution in class next Friday (10 ∼ 20 minutes). Emphasized on surprises and lesson learned No need to be comprehensive HW1 won’t be accepted after his presentation S. Cheng (OU-Tulsa) Convolutional Neural Networks Jan 2017 4 / 198 Review Review In the last class, we discussed BP Weight initialization Batch normalization Dropout More optimization tricks Nesterov accelerated gradient descent RMSProp Adam S. Cheng (OU-Tulsa) Convolutional Neural Networks Jan 2017 5 / 198 Babysitting your learning job Debugging optimizer Today Left out from last lecture: some remarks on babysitting your training process Convolutional neural network (CNN) S. Cheng (OU-Tulsa) Convolutional Neural Networks Jan 2017 6 / 198 Babysitting your learning job Debugging optimizer Debugging optimizer Lecture 5 - 20 Jan 2016 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 20 Jan 2016 75 Double check that the loss is reasonable: crank up regularization loss went up, good. (sanity check) S. Cheng (OU-Tulsa) Convolutional Neural Networks Jan 2017 7 / 198 Babysitting your learning job Debugging optimizer Debugging optimizer Lecture 5 - 20 Jan 2016 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 20 Jan 2016 76 Lets try to train now… Tip: Make sure that you can overfit very small portion of the training data The above code: - take the first 20 examples from CIFAR-10 - turn off regularization (reg = 0.0) - use simple vanilla ‘sgd’ S. Cheng (OU-Tulsa) Convolutional Neural Networks Jan 2017 8 / 198 Babysitting your learning job Debugging optimizer Debugging optimizer Lecture 5 - 20 Jan 2016 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 20 Jan 2016 77 Lets try to train now… Tip: Make sure that you can overfit very small portion of the training data Very small loss, train accuracy 1.00, nice! S. Cheng (OU-Tulsa) Convolutional Neural Networks Jan 2017 9 / 198 Babysitting your learning job Debugging optimizer Debugging optimizer Lecture 5 - 20 Jan 2016 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 20 Jan 2016 78 Lets try to train now… I like to start with small regularization and find learning rate that makes the loss go down. S. Cheng (OU-Tulsa) Convolutional Neural Networks Jan 2017 10 / 198 Babysitting your learning job Debugging optimizer Debugging optimizer Lecture 5 - 20 Jan 2016 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 20 Jan 2016 79 Lets try to train now… I like to start with small regularization and find learning rate that makes the loss go down. Loss barely changing S. Cheng (OU-Tulsa) Convolutional Neural Networks Jan 2017 11 / 198 Babysitting your learning job Debugging optimizer Debugging optimizer Lecture 5 - 20 Jan 2016 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 20 Jan 2016 80 Lets try to train now… I like to start with small regularization and find learning rate that makes the loss go down. loss not going down: learning rate too low Loss barely changing: Learning rate is probably too low S. Cheng (OU-Tulsa) Convolutional Neural Networks Jan 2017 12 / 198 Babysitting your learning job Debugging optimizer Debugging optimizer Lecture 5 - 20 Jan 2016 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 20 Jan 2016 81 Lets try to train now… I like to start with small regularization and find learning rate that makes the loss go down. loss not going down: learning rate too low Loss barely changing: Learning rate is probably too low Notice train/val accuracy goes to 20% though, what’s up with that? (remember this is softmax) S. Cheng (OU-Tulsa) Convolutional Neural Networks Jan 2017 13 / 198 Babysitting your learning job Debugging optimizer Debugging optimizer Lecture 5 - 20 Jan 2016 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 20 Jan 2016 82 Lets try to train now… I like to start with small regularization and find learning rate that makes the loss go down. loss not going down: learning rate too low Okay now lets try learning rate 1e6. What could possibly go wrong? S. Cheng (OU-Tulsa) Convolutional Neural Networks Jan 2017 14 / 198 Babysitting your learning job Debugging optimizer Debugging optimizer Lecture 5 - 20 Jan 2016 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 20 Jan 2016 83 cost: NaN almost always means high learning rate... Lets try to train now… I like to start with small regularization and find learning rate that makes the loss go down. loss not going down: learning rate too low loss exploding: learning rate too high S. Cheng (OU-Tulsa) Convolutional Neural Networks Jan 2017 15 / 198