P IN RACTICE SECOND EDITION Ian Miell Aidan Hobson Sayers M A N N I N G Praise for the First Edition A deluge of practical advice about applying Docker to problems you have right now. —From the Foreword to the first edition by Ben Firshman, Docker, Inc. Filled with 4-star recipes! —Chad Davis, SolidFire You'll love Docker after reading this book. —José San Leandro, OSOCO Packed with Docker tricks of the developer trade. —Kirk Brattkus, Net Effect Technologies Extremely good sets of tips for using docker. Really useful and pragmatic and addresses real world docker issues. —Amazon customer Easy to read and follow. I have a much better understanding of the internal working of Docker after reading this book. —Amazon customer Docker in Practice S E ECOND DITION IAN MIELL AIDAN HOBSON SAYERS MANNING Shelter Island For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: [email protected] ©2019 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. Manning Publications Co. Development editor: Jenny Stout 20 Baldwin Road Review editor: Ivan Martinovic´ PO Box 761 Project manager: Vincent Nordhaus Shelter Island, NY 11964 Copyeditor: Andy Carroll Proofreader: Keri Hales Technical proofreader: Jose San Leandro Typesetter: Gordan Salinovic Cover designer: Marija Tudor ISBN 9781617294808 Printed in the United States of America 1 2 3 4 5 6 7 8 9 10 – SP – 24 23 22 21 20 19 contents preface xiii acknowledgments xv about this book xvi about the cover illustration xix P 1 D ..........................................1 ART OCKER FUNDAMENTALS 1 Discovering Docker 3 1.1 The what and why of Docker 4 What is Docker? 5 ■ What is Docker good for? 7 ■ Key concepts 8 1.2 Building a Docker application 10 Ways to create a new Docker image 11 ■ Writing a Dockerfile 12 Building a Docker image 13 ■ Running a Docker container 14 Docker layering 16 2 Understanding Docker: Inside the engine room 19 2.1 Docker’s architecture 20 2.2 The Docker daemon 21 TECHNIQUE 1 Open your Docker daemon to the world 22 TECHNIQUE 2 Running containers as daemons 24 TECHNIQUE 3 Moving Docker to a different partition 27 v vi CONTENTS 2.3 The Docker client 28 TECHNIQUE 4 Using socat to monitor Docker API traffic 28 TECHNIQUE 5 Using Docker in your browser 32 TECHNIQUE 6 Using ports to connect to containers 34 TECHNIQUE 7 Allowing container communication 35 TECHNIQUE 8 Linking containers for port isolation 37 2.4 Docker registries 39 TECHNIQUE 9 Setting up a local Docker registry 40 2.5 The Docker Hub 41 TECHNIQUE 10 Finding and running a Docker image 42 P 2 D ...................................45 ART OCKER AND DEVELOPMENT 3 Using Docker as a lightweight virtual machine 47 3.1 From VM to container 48 TECHNIQUE 11 Converting your VM to a container 48 TECHNIQUE 12 A host-like container 51 TECHNIQUE 13 Splitting a system into microservice containers 54 TECHNIQUE 14 Managing the startup of your container’s services 57 3.2 Saving and restoring your work 59 TECHNIQUE 15 The “save game” approach: Cheap source control 60 TECHNIQUE 16 Docker tagging 62 TECHNIQUE 17 Sharing images on the Docker Hub 65 TECHNIQUE 18 Referring to a specific image in builds 67 3.3 Environments as processes 68 TECHNIQUE 19 The “save game” approach: Winning at 2048 68 4 Building images 71 4.1 Building images 71 TECHNIQUE 20 Injecting files into your image using ADD 72 TECHNIQUE 21 Rebuilding without the cache 75 TECHNIQUE 22 Busting the cache 76 TECHNIQUE 23 Intelligent cache-busting using build-args 78 TECHNIQUE 24 Intelligent cache-busting using the ADD directive 81 TECHNIQUE 25 Setting the right time zone in your containers 84 TECHNIQUE 26 Locale management 86 TECHNIQUE 27 Stepping through layers with the image-stepper 90 TECHNIQUE 28 Onbuild and golang 94 CONTENTS vii 5 Running containers 97 5.1 Running containers 97 TECHNIQUE 29 Running GUIs within Docker 98 TECHNIQUE 30 Inspecting containers 100 TECHNIQUE 31 Cleanly killing containers 101 TECHNIQUE 32 Using Docker Machine to provision Docker hosts 103 TECHNIQUE 33 Wildcard DNS 107 5.2 Volumes—a persistent problem 108 TECHNIQUE 34 Docker volumes: Problems of persistence 108 TECHNIQUE 35 Distributed volumes with Resilio Sync 110 TECHNIQUE 36 Retaining your container’s bash history 112 TECHNIQUE 37 Data containers 114 TECHNIQUE 38 Remote volume mounting using SSHFS 116 TECHNIQUE 39 Sharing data over NFS 119 TECHNIQUE 40 Dev tools container 121 6 Day-to-day Docker 124 6.1 Staying ship-shape 124 TECHNIQUE 41 Running Docker without sudo 125 TECHNIQUE 42 Housekeeping containers 125 TECHNIQUE 43 Housekeeping volumes 127 TECHNIQUE 44 Detaching containers without stopping them 129 TECHNIQUE 45 Using Portainer to manage your Docker daemon 130 TECHNIQUE 46 Generating a dependency graph of your Docker images 131 TECHNIQUE 47 Direct action: Executing commands on your container 133 TECHNIQUE 48 Are you in a Docker container? 134 7 Configuration management: Getting your house in order 137 7.1 Configuration management and Dockerfiles 138 TECHNIQUE 49 Creating reliable bespoke tools with ENTRYPOINT 138 TECHNIQUE 50 Avoiding package drift by specifying versions 140 TECHNIQUE 51 Replacing text with perl -p -i -e 142 TECHNIQUE 52 Flattening images 143 TECHNIQUE 53 Managing foreign packages with Alien 145 7.2 Traditional configuration management tools with Docker 148 TECHNIQUE 54 Traditional: Using make with Docker 148 TECHNIQUE 55 Building images with Chef Solo 151 viii CONTENTS 7.3 Small is beautiful 155 TECHNIQUE 56 Tricks for making an image smaller 155 TECHNIQUE 57 Tiny Docker images with BusyBox and Alpine 157 TECHNIQUE 58 The Go model of minimal containers 159 TECHNIQUE 59 Using inotifywait to slim containers 162 TECHNIQUE 60 Big can be beautiful 164 P 3 D D O .........................................167 ART OCKER AND EV PS 8 Continuous integration: Speeding up your development pipeline 169 8.1 Docker Hub automated builds 170 TECHNIQUE 61 Using the Docker Hub workflow 170 8.2 More efficient builds 173 TECHNIQUE 62 Speeding up I/O-intensive builds with eatmydata 173 TECHNIQUE 63 Setting up a package cache for faster builds 175 TECHNIQUE 64 Headless Chrome in a container 178 TECHNIQUE 65 Running Selenium tests inside Docker 180 8.3 Containerizing your CI process 185 TECHNIQUE 66 Running the Jenkins master within a Docker container 185 TECHNIQUE 67 Containing a complex development environment 188 TECHNIQUE 68 Scaling your CI with Jenkins’ Swarm plugin 192 TECHNIQUE 69 Upgrading your containerized Jenkins server safely 195 9 Continuous delivery: A perfect fit for Docker principles 200 9.1 Interacting with other teams in the CD pipeline 201 TECHNIQUE 70 The Docker contract: Reducing friction 201 9.2 Facilitating deployment of Docker images 204 TECHNIQUE 71 Manually mirroring registry images 204 TECHNIQUE 72 Delivering images over constrained connections 206 TECHNIQUE 73 Sharing Docker objects as TAR files 208 9.3 Configuring your images for environments 210 TECHNIQUE 74 Informing your containers with etcd 210 9.4 Upgrading running containers 214 TECHNIQUE 75 Using confd to enable zero-downtime switchovers 214 CONTENTS ix 10 Network simulation: Realistic environment testing without the pain 220 10.1 Container communication: Beyond manual linking 221 TECHNIQUE 76 A simple Docker Compose cluster 221 TECHNIQUE 77 A SQLite server using Docker Compose 225 10.2 Using Docker to simulate real-world networking 229 TECHNIQUE 78 Simulating troublesome networks with Comcast 230 TECHNIQUE 79 Simulating troublesome networks with Blockade 233 10.3 Docker and virtual networks 237 TECHNIQUE 80 Creating another Docker virtual network 238 TECHNIQUE 81 Setting up a substrate network with Weave 241 P 4 O ART RCHESTRATION FROM A SINGLE MACHINE TO ........................................................247 THE CLOUD 11 A primer on container orchestration 249 11.1 Simple single-host Docker 251 TECHNIQUE 82 Managing your host’s containers with systemd 251 TECHNIQUE 83 Orchestrating the startup of your host’s containers 254 11.2 Manual multi-host Docker 257 TECHNIQUE 84 Manual multi-host Docker with Helios 257 11.3 Service discovery: What have we here? 263 TECHNIQUE 85 Using Consul to discover services 264 TECHNIQUE 86 Automatic service registration with Registrator 271 12 The data center as an OS with Docker 275 12.1 Multi-host Docker 276 TECHNIQUE 87 A seamless Docker cluster with swarm mode 276 TECHNIQUE 88 Using a Kubernetes cluster 280 TECHNIQUE 89 Accessing the Kubernetes API from within a pod 285 TECHNIQUE 90 Using OpenShift to run AWS APIs locally 289 TECHNIQUE 91 Building a framework on Mesos 294 TECHNIQUE 92 Micromanaging Mesos with Marathon 302