Deep Learning AMI Developer Guide Deep Learning AMI Developer Guide Deep Learning AMI: Developer Guide Copyright © 2023 Amazon Web Services, Inc. and/or its affiliates. All rights reserved. Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon. Deep Learning AMI Developer Guide Table of Contents What Is the AWS Deep Learning AMI? ................................................................................................... 1 About This Guide....................................................................................................................... 1 Prerequisites.............................................................................................................................. 1 Example Uses............................................................................................................................ 1 Features.................................................................................................................................... 2 Preinstalled Frameworks..................................................................................................... 2 Preinstalled GPU Software.................................................................................................. 2 Model Serving and Visualization .......................................................................................... 2 Getting Started .................................................................................................................................. 3 How to Get Started with the DLAMI ............................................................................................. 3 DLAMI Selection......................................................................................................................... 3 CUDA Installations and Framework Bindings .......................................................................... 4 Base................................................................................................................................. 4 Conda............................................................................................................................... 5 Architecture....................................................................................................................... 6 OS .................................................................................................................................... 6 Instance Selection...................................................................................................................... 6 Pricing.............................................................................................................................. 7 Region Availability.............................................................................................................. 8 GPU.................................................................................................................................. 8 CPU.................................................................................................................................. 8 Inferentia.......................................................................................................................... 9 Habana............................................................................................................................. 9 Framework Support Policy ................................................................................................................. 11 Supported Frameworks............................................................................................................. 11 Frequently Asked Questions....................................................................................................... 11 What framework versions get security patches? ................................................................... 12 What images does AWS publish when new framework versions are released? ............................ 12 What images get new SageMaker/AWS features? .................................................................. 12 How is current version defined in the Supported Frameworks table? ....................................... 12 What if I am running a version that is not in the Supported Frameworks table? ......................... 12 Do DLAMIs support previous versions of TensorFlow? ............................................................ 12 How can I find the latest patched image for a supported framework version? ........................... 12 How frequently are new images released? ........................................................................... 13 Will my instance be patched in place while my workload is running? ....................................... 13 What happens when a new patched or updated framework version is available? ....................... 13 Are dependencies updated without changing the framework version? ..................................... 13 When does active support for my framework version end? ..................................................... 13 Will images with framework versions that are no longer actively maintained be patched? ........... 14 How do I use an older framework version? .......................................................................... 14 How do I stay up-to-date with support changes in frameworks and their versions? .................... 14 Do I need a commercial license to use the Anaconda Repository? ............................................ 15 Launching a DLAMI .......................................................................................................................... 16 Step 1: Launch a DLAMI ............................................................................................................ 16 Retrieve the DLAMI ID ...................................................................................................... 16 Launch from Amazon EC2 Console ..................................................................................... 17 Step 2: Connect to the DLAMI ................................................................................................... 18 Step 3: Test Your DLAMI ........................................................................................................... 18 Step 4: Manage Your DLAMI Instance .......................................................................................... 18 Clean Up................................................................................................................................. 18 Jupyter Setup.......................................................................................................................... 19 Secure Jupyter................................................................................................................. 19 Start Server..................................................................................................................... 20 Configure Client............................................................................................................... 20 iii Deep Learning AMI Developer Guide Log in to the Jupyter notebook server ................................................................................ 21 Using a DLAMI ................................................................................................................................. 25 Conda DLAMI........................................................................................................................... 25 Introduction to the Deep Learning AMI with Conda .............................................................. 25 Log in to Your DLAMI ....................................................................................................... 25 Start the TensorFlow Environment ..................................................................................... 26 Switch to the PyTorch Python 3 Environment ...................................................................... 27 Switch to the MXNet Python 3 Environment ........................................................................ 27 Removing Environments.................................................................................................... 28 Base DLAMI............................................................................................................................. 28 Using the Deep Learning Base AMI..................................................................................... 28 Configuring CUDA Versions ................................................................................................ 29 Jupyter Notebooks................................................................................................................... 29 Navigating the Installed Tutorials ....................................................................................... 29 Switching Environments with Jupyter ................................................................................. 30 Tutorials.................................................................................................................................. 30 10 Minute Tutorials .......................................................................................................... 30 Activating Frameworks...................................................................................................... 31 Debugging and Visualization.............................................................................................. 44 Distributed Training.......................................................................................................... 48 Elastic Fabric Adapter ....................................................................................................... 65 GPU Monitoring and Optimization...................................................................................... 77 AWS Inferentia................................................................................................................. 86 Graviton DLAMI.............................................................................................................. 101 Habana DLAMI............................................................................................................... 108 Inference....................................................................................................................... 109 Using Frameworks with ONNX ......................................................................................... 113 Model Serving ................................................................................................................ 121 Upgrading Your DLAMI.................................................................................................................... 128 DLAMI Upgrade...................................................................................................................... 128 Software Updates................................................................................................................... 128 Security......................................................................................................................................... 130 Data Protection...................................................................................................................... 130 Identity and Access Management .............................................................................................. 131 Authenticating With Identities.......................................................................................... 131 Managing Access Using Policies ........................................................................................ 133 IAM with Amazon EMR .................................................................................................... 135 Logging and Monitoring.......................................................................................................... 135 Usage Tracking............................................................................................................... 135 Compliance Validation............................................................................................................. 135 Resilience.............................................................................................................................. 136 Infrastructure Security............................................................................................................. 136 Related Information........................................................................................................................ 137 Forums.................................................................................................................................. 137 Blogs.................................................................................................................................... 137 FAQ...................................................................................................................................... 137 Release Notes for DLAMI ................................................................................................................. 140 Single-framework DLAMI......................................................................................................... 140 Multi-framework DLAMI.......................................................................................................... 141 GPU DLAMI............................................................................................................................ 141 Habana DLAMI....................................................................................................................... 142 Base DLAMI........................................................................................................................... 142 DLAMI Deprecation Notices.............................................................................................................. 143 Document History.......................................................................................................................... 145 AWS glossary................................................................................................................................. 148 iv Deep Learning AMI Developer Guide About This Guide What Is the AWS Deep Learning AMI? Welcome to the User Guide for the AWS Deep Learning AMI. The AWS Deep Learning AMI (DLAMI) is your one-stop shop for deep learning in the cloud. This customized machine instance is available in most Amazon EC2 regions for a variety of instance types, from a small CPU-only instance to the latest high-powered multi-GPU instances. It comes preconfigured with NVIDIA CUDA and NVIDIA cuDNN, as well as the latest releases of the most popular deep learning frameworks. About This Guide This guide will help you launch and use the DLAMI. It covers several use cases that are common for deep learning, for both training and inference. Choosing the right AMI for your purpose and the kind of instances you may prefer is also covered. The DLAMI comes with several tutorials for each of the frameworks. It also has tutorials on distributed training, debugging, using AWS Inferentia, and other key concepts. You will find instructions on how to configure Jupyter to run the tutorials in your browser. Prerequisites You should be familiar with command line tools and basic Python to successfully run the DLAMI. Tutorials on how to use each framework are provided by the frameworks themselves, however, this guide can show you how to activate each one and find the appropriate tutorials to get started. Example DLAMI Uses Learning about deep learning: The DLAMI is a great choice for learning or teaching machine learning and deep learning frameworks. It takes the headache away from troubleshooting the installations of each framework and getting them to play along on the same computer. The DLAMI comes with a Jupyter notebook and makes it easy to run the tutorials provided by the frameworks for people new to machine learning and deep learning. App development: If you're an app developer and are interested in using deep learning to make your apps utilize the latest advances in AI, the DLAMI is the perfect test bed for you. Each framework comes with tutorials on how to get started with deep learning, and many of them have model zoos that make it easy to try out deep learning without having to create the neural networks yourself or to do any of the model training. Some examples show you how to build an image detection application in just a few minutes, or how to build a speech recognition app for your own chatbot. Machine learning and data analytics: If you're a data scientist or interested in processing your data with deep learning, you'll find that many of the frameworks have support for R and Spark. You will find tutorials on how to do simple regressions, all the way up to building scalable data processing systems for personalization and predictions systems. Research: If you're a researcher and want to try out a new framework, test out a new model, or train new models, the DLAMI and AWS capabilities for scale can alleviate the pain of tedious installations and management of multiple training nodes. 1 Deep Learning AMI Developer Guide Features Note While your initial choice might be to upgrade your instance type up to a larger instance with more GPUs (up to 8), you can also scale horizontally by creating a cluster of DLAMI instances. Check out Related Information (p. 137) for more information on cluster builds. Features of the DLAMI Preinstalled Frameworks There are currently two primary flavors of the DLAMI with other variations related to the operating system (OS) and software versions: • Deep Learning AMI with Conda (p. 5) - frameworks installed separately using conda packages and separate Python environments • Deep Learning Base AMI (p. 4) - no frameworks installed; only NVIDIA CUDA and other dependencies The Deep Learning AMI with Conda uses conda environments to isolate each framework, so you can switch between them at will and not worry about their dependencies conflicting. This is the full list of supported frameworks by Deep Learning AMI with Conda: • Apache MXNet (Incubating) • PyTorch • TensorFlow 2 Note We no longer include the CNTK, Caffe, Caffe2, Theano, Chainer, or Keras Conda environments in the AWS Deep Learning AMI starting with the v28 release. Previous releases of the AWS Deep Learning AMI that contain these environments will continue to be available. However, we will only provide updates to these environments if there are security fixes published by the open source community for these frameworks. Preinstalled GPU Software Even if you use a CPU-only instance, the DLAMI will have NVIDIA CUDA and NVIDIA cuDNN. The installed software is the same regardless of the instance type. Keep in mind that GPU-specific tools only work on an instance that has at least one GPU. More information on this is covered in the Selecting the Instance Type for DLAMI (p. 6). For more information on CUDA Installation, see the CUDA Installations and Framework Bindings (p. 4). Model Serving and Visualization Deep Learning AMI with Conda comes preinstalled with two kinds of model servers, one for MXNet and one for TensorFlow, as well as TensorBoard, for model visualizations. • Model Server for Apache MXNet (MMS) (p. 122) • TensorFlow Serving (p. 124) • TensorBoard (p. 46) 2 Deep Learning AMI Developer Guide How to Get Started with the DLAMI Getting Started How to Get Started with the DLAMI This guide includes tips about picking the DLAMI that's right for you, selecting an instance type that fits your use case and budget, and Related Information (p. 137) that describes custom setups that may be of interest. If you're new to using AWS or using Amazon EC2, start with the Deep Learning AMI with Conda (p. 5). If you're familiar with Amazon EC2 and other AWS services like Amazon EMR, Amazon EFS, or Amazon S3, and are interested in integrating those services for projects that need distributed training or inference, then check out Related Information (p. 137) to see if one fits your use case. We recommend that you check out Choosing Your DLAMI (p. 3) to get an idea of which instance type might be best for your application. Another option is this quick tutorial: Launch a AWS Deep Learning AMI (in 10 minutes). Next Step Choosing Your DLAMI (p. 3) Choosing Your DLAMI We offer a range of DLAMI options. To help you select the correct DLAMI for your use case, we group images by the hardware type or functionality for which they were developed. Our top level groupings are: • DLAMI Type: CUDA versus Base versus Single-Framework versus Multi-Framework (Conda DLAMI) • Compute Architecture: x86-based versus Arm-based AWS Graviton • Processor Type: GPU versus CPU versus Inferentia versus Habana • SDK: CUDA versus AWS Neuron versus SynapsesAI • OS: Amazon Linux versus Ubuntu The rest of the topics in this guide help further inform you and go into more details. Topics • CUDA Installations and Framework Bindings (p. 4) • Deep Learning Base AMI (p. 4) • Deep Learning AMI with Conda (p. 5) • DLAMI CPU Architecture Options (p. 6) • DLAMI Operating System Options (p. 6) Next Up Deep Learning AMI with Conda (p. 5) 3 Deep Learning AMI Developer Guide CUDA Installations and Framework Bindings CUDA Installations and Framework Bindings While deep learning is all pretty cutting edge, each framework offers "stable" versions. These stable versions may not work with the latest CUDA or cuDNN implementation and features. Your use case and the features you require can help you choose a framework. If you are not sure, then use the latest Deep Learning AMI with Conda. It has official pip binaries for all frameworks with CUDA 10, using whichever most recent version is supported by each framework. If you want the latest versions, and to customize your deep learning environment, use the Deep Learning Base AMI. Look at our guide on Stable Versus Release Candidates (p. 5) for further guidance. Choose a DLAMI with CUDA The Deep Learning Base AMI (p. 4) has all available CUDA 11 series, including 11.0, 11.1, and 11.2. The Deep Learning AMI with Conda (p. 5) has all available CUDA 11 series, including 11.0, 11.1, and 11.2. Note We no longer include the CNTK, Caffe, Caffe2, Theano, Chainer, or Keras Conda environments in the AWS Deep Learning AMI starting with the v28 release. Previous releases of the AWS Deep Learning AMI that contain these environments continue to be available. However, we only provide updates to these environments if there are security fixes published by the open-source community for these frameworks. For specific framework version numbers, see the Release Notes for DLAMI (p. 140) Choose this DLAMI type or learn more about the different DLAMIs with the Next Up option. Choose one of the CUDA versions and review the full list of DLAMIs that have that version in the Appendix, or learn more about the different DLAMIs with the Next Up option. Next Up Deep Learning Base AMI (p. 4) Related Topics • For instructions on switching between CUDA versions, refer to the Using the Deep Learning Base AMI (p. 28) tutorial. Deep Learning Base AMI The Deep Learning Base AMI is like an empty canvas for deep learning. It comes with everything you need up until the point of the installation of a particular framework, and has your choice of CUDA versions. Why to Choose the Base DLAMI This AMI group is useful for project contributors who want to fork a deep learning project and build the latest. It's for someone who wants to roll their own environment with the confidence that the latest NVIDIA software is installed and working so they can focus on picking which frameworks and versions they want to install. Choose this DLAMI type or learn more about the different DLAMIs with the Next Up option. Next Up 4 Deep Learning AMI Developer Guide Conda DLAMI with Conda Related Topics • Using the Deep Learning Base AMI Deep Learning AMI with Conda The Conda DLAMI uses conda virtual environments. These environments are configured to keep the different framework installations separate and streamline switching between frameworks. This is great for learning and experimenting with all of the frameworks the DLAMI has to offer. Most users find that the new Deep Learning AMI with Conda is perfect for them. These AMIs are the primary DLAMIs. They are updated often with the latest versions from the frameworks, and have the latest GPU drivers and software. They are generally referred to as the AWS Deep Learning AMI in most documents. • The Ubuntu 18.04 DLAMI has the following frameworks: Apache MXNet (Incubating), PyTorch, and TensorFlow 2. • The Amazon Linux 2 DLAMI has the following frameworks: Apache MXNet (Incubating), PyTorch, and TensorFlow 2. Note We no longer include the CNTK, Caffe, Caffe2, Theano, Chainer, and Keras Conda environments in the AWS Deep Learning AMI starting with the v28 release. Previous releases of the AWS Deep Learning AMI that contain these environments continue to be available. However, we only provide updates to these environments if there are security fixes published by the open-source community for these frameworks. Stable Versus Release Candidates The Conda AMIs use optimized binaries of the most recent formal releases from each framework. Release candidates and experimental features are not to be expected. The optimizations depend on the framework's support for acceleration technologies like Intel's MKL DNN, which speeds up training and inference on C5 and C4 CPU instance types. The binaries are also compiled to support advanced Intel instruction sets including but not limited to AVX, AVX-2, SSE4.1, and SSE4.2. These accelerate vector and floating point operations on Intel CPU architectures. Additionally, for GPU instance types, the CUDA and cuDNN are updated with whichever version the latest official release supports. The Deep Learning AMI with Conda automatically installs the most optimized version of the framework for your Amazon EC2 instance upon the framework's first activation. For more information, refer to Using the Deep Learning AMI with Conda (p. 25). If you want to install from source, using custom or optimized build options, the Deep Learning Base AMI (p. 4)s might be a better option for you. Python 2 Deprecation The Python open source community has officially ended support for Python 2 on January 1, 2020. The TensorFlow and PyTorch community have announced that the TensorFlow 2.1 and PyTorch 1.4 releases are the last ones supporting Python 2. Previous releases of the DLAMI (v26, v25, etc) that contain Python 2 Conda environments continue to be available. However, we provide updates to the Python 2 Conda environments on previously published DLAMI versions only if there are security fixes published by the open-source community for those versions. DLAMI releases with the latest versions of the TensorFlow and PyTorch frameworks do not contain the Python 2 Conda environments. 5 Deep Learning AMI Developer Guide Architecture CUDA Support Specific CUDA version numbers can be found in the GPU DLAMI release notes. Next Up DLAMI CPU Architecture Options (p. 6) Related Topics • For a tutorial on using a Deep Learning AMI with Conda, see the Using the Deep Learning AMI with Conda (p. 25) tutorial. DLAMI CPU Architecture Options AWS Deep Learning AMIs are offered with either x86-based or Arm-based AWS Graviton2 CPU architectures. Choose one of the Graviton GPU DLAMIs to work with an Arm-based CPU architecture. All other GPU DLAMIs are currently x86-based. • AWS Deep Learning AMI Graviton GPU CUDA 11.4 (Ubuntu 20.04) • AWS Deep Learning AMI Graviton GPU TensorFlow 2.6 (Ubuntu 20.04) • AWS Deep Learning AMI Graviton GPU PyTorch 1.10 (Ubuntu 20.04) For information about getting started with the Graviton GPU DLAMI, see The Graviton DLAMI (p. 101). For more details on available instance types, see Selecting the Instance Type for DLAMI (p. 6). Next Up DLAMI Operating System Options (p. 6) DLAMI Operating System Options DLAMIs are offered in the following operating systems. • Amazon Linux 2 • Ubuntu 20.04 • Ubuntu 18.04 Older versions of operating systems are available on deprecated DLAMIs. For more information on DLAMI deprecation, see Deprecations for DLAMI Before choosing a DLAMI, assess what instance type you need and identify your AWS Region. Next Up Selecting the Instance Type for DLAMI (p. 6) Selecting the Instance Type for DLAMI See the AWS Deep Learning AMI Catalog for recommended Amazon EC2 instance families that are compatible with specific DLAMIs. 6