THE EXPERT’S VOICE® IN AUTOMATION Using Chef with Microsoft Azure — Stuart Preston Using Chef with Microsoft Azure Stuart Preston Using Chef with Microsoft Azure Stuart Preston London, United Kingdom ISBN-13 (pbk): 978-1-4842-1477-0 ISBN-13 (electronic): 978-1-4842-1476-3 DOI 10.1007/978-1-4842-1476-3 Library of Congress Control Number: 2016941179 Copyright © 2016 by Stuart Preston This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher’s location, in its current version, and permission for use must always be obtained from Springer. Permissions for use may be obtained through RightsLink at the Copyright Clearance Center. Violations are liable to prosecution under the respective Copyright Law. Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Managing Director: Welmoed Spahr Lead Editor: Gwenan Spearing Technical Reviewer: John Fitzpatrick Editorial Board: Steve Anglin, Pramila Balen, Louise Corrigan, Jim DeWolf, Jonathan Gennick, Robert Hutchinson, Celestin Suresh John, Nikhil Karkal, James Markham, Susan McDermott, Matthew Moodie, Douglas Pundick, Ben Renow-Clarke, Gwenan Spearing Coordinating Editor: Melissa Maldonado Copy Editor: Karen Jameson Compositor: SPi Global Indexer: SPi Global Artist: SPi Global Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail [email protected] , or visit w ww.springer.com . Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation. For information on translations, please e-mail r [email protected] , or visit w ww.apress.com . Apress and friends of ED books may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Special Bulk Sales–eBook Licensing web page at w ww.apress.com/bulk-sales . Any source code or other supplementary material referenced by the author in this text is available to readers at w ww.apress.com . For detailed information about how to locate your book’s source code, go to www.apress.com/source-code/ . Printed on acid-free paper Contents at a Glance About the Author .....................................................................................................xi About the Technical Reviewer ...............................................................................xiii Introduction .............................................................................................................xv ■ Chapter 1: Confi guration Management using Chef ................................................1 ■ Chapter 2: Microsoft Azure Terminology and Concepts ......................................29 ■ Chapter 3: Chef Azure VM Extensions .................................................................55 ■ Chapter 4: Using Chef Provisioning to Provision Machines .................................71 ■ Chapter 5: Advanced Chef Provisioning Techniques .........................................101 ■ Chapter 6: Integrating Quality Tooling into the Chef Development Life Cycle .........131 ■ Chapter 7: Chef Concepts in the Real World ......................................................163 ■ Chapter 8: Pulling It All Together: Continuous Provisioning with Chef and Azure ..................................................................................................173 ■ Appendix A: Further Resources .........................................................................221 Index .....................................................................................................................225 iii Contents About the Author .....................................................................................................xi About the Technical Reviewer ...............................................................................xiii Introduction .............................................................................................................xv ■ Chapter 1: Confi guration Management using Chef ................................................1 The Purpose and Principles of Automated Provisioning and Confi guration Management ..............................................................................................2 Chef Architecture ..............................................................................................................3 Chef Client and Chef Server-supported Platforms ..................................................................................5 Getting Ready for Chef Development ......................................................................................................6 Developing Your First Recipe Using Chef .......................................................................19 Code Editors .........................................................................................................................................19 Initializing a Chef Repository ................................................................................................................20 Modifying and Running the Default Recipe ..........................................................................................21 Getting Started with Hosted Chef ...................................................................................23 Summary ........................................................................................................................28 ■ Chapter 2: Microsoft Azure Terminology and Concepts ......................................29 Deploying to the Microsoft Azure Platform .....................................................................29 Subscriptions, Tenants, and Regions ..............................................................................31 Subscriptions ........................................................................................................................................31 Tenants .................................................................................................................................................32 Regions .................................................................................................................................................33 v ■ CONTENTS Managing Azure from the Command Line ......................................................................35 Installing the Tools (Windows) ..............................................................................................................36 Installing the Tools (Mac OS X) .............................................................................................................38 Installing the Tools (Linux) ....................................................................................................................39 Logging In and Verifying Command-line Tools Connectivity .................................................................40 Azure Resource Groups ..................................................................................................43 Azure Resource Manager Templates ..............................................................................44 Resource Manager Template Structure ................................................................................................44 Example Resource Manager Template .................................................................................................45 Expressions and Functions ...................................................................................................................52 Summary ........................................................................................................................53 ■ Chapter 3: Chef Azure VM Extensions .................................................................55 What Are Azure VM Extensions? .....................................................................................55 Introducing Chef VM Extensions .....................................................................................56 Chef Azure VM Extension Compatibility ................................................................................................57 Listing the Available Chef VM Extension Versions from the VM Extension Gallery ...............................57 Adding a Chef VM Extension to an Existing Virtual Machine .................................................................61 Validating a Chef VM Extension is successfully installed at the Command Line ..................................63 Removing a Chef VM Extension from a Virtual Machine .......................................................................66 Installing a Chef VM Extension at the Command Line Using Azure Resource Manager Template Language ...............................................................................................................67 Summary ........................................................................................................................69 ■ Chapter 4: Using Chef Provisioning to Provision Machines .................................71 About Chef Provisioning on Azure ..................................................................................71 Installation and Confi guration ........................................................................................72 Installing the Chef Provisioning for Azure Resource Manager Ruby Gem ............................................72 Authenticating to Azure Resource Manager .........................................................................................74 Confi guring the Application and Service Principal ...............................................................................75 Confi guring Chef Provisioning for Authentication .................................................................................81 Preparing the Chef-Repo ......................................................................................................................82 vi ■ CONTENTS Chef Provisioning Recipes ..............................................................................................83 Confi guring the Provisioning Node as a Chef Client .............................................................................84 Executing the Provisioning Recipe .......................................................................................................85 Chef Provisioning a Windows VM with Remote Desktop Enabled ........................................................87 Chef Provisioning and the Chef VM Extensions ....................................................................................94 Destroying Azure Resources .................................................................................................................97 Summary ........................................................................................................................99 ■ Chapter 5: Advanced Chef Provisioning Techniques .........................................101 Explaining VM Image Naming within Azure Resource Manager JSON .........................101 Identifying and Retrieving VM Images ................................................................................................103 Using Azure Key Vault to Store Secrets ........................................................................105 Azure Key Vault ARM Template ...........................................................................................................106 Retrieving the Object ID for an Azure Active Directory User ...............................................................108 Azure Key Vault Provisioning Recipe ..................................................................................................109 Creating a Windows Server with WinRM Securely Enabled via Key Vault ....................111 Creating a Self-signed Certifi cate ......................................................................................................111 Uploading the Certifi cate to Key Vault ................................................................................................112 Provisioning a WinRM-Enabled Windows Server ................................................................................114 Verifying WinRM Status ......................................................................................................................121 Creating Other PaaS Resources via Chef Provisioning and Resource Explorer ............123 Creating a Dummy Resource ..............................................................................................................124 Viewing the Resource in Resource Explorer .......................................................................................127 Extracting the Template ......................................................................................................................128 Running a Custom Deployment ..........................................................................................................129 Summary ......................................................................................................................130 ■ Chapter 6: Integrating Quality Tooling into the Chef Development Life Cycle .......131 Cookbook Linting ..........................................................................................................131 Using RuboCop ...................................................................................................................................132 Using FoodCritic .................................................................................................................................140 vii ■ CONTENTS Cookbook Testing .........................................................................................................145 Using ChefSpec ..................................................................................................................................146 Using Test Kitchen and InSpec with Azure Resource Manager ....................................153 Installing the Azure Resource Manager Driver for Test Kitchen ..........................................................154 Confi guring the Credentials File .........................................................................................................154 Confi guring Test Kitchen within a Chef Repo ......................................................................................155 Summary ......................................................................................................................162 ■ Chapter 7: Chef Concepts in the Real World ......................................................163 Avoid Using the _default Environment .........................................................................163 Use Chef Environments to Refl ect Your Internal Release Processes ............................164 Listing Existing Environments ............................................................................................................164 Creating New Environments ...............................................................................................................164 Changing the Environment for a Node ................................................................................................166 Searching for Servers in an Existing Environment .............................................................................168 Controlling Releases through Environments Using the Environment and Role Patterns ................................................................................................................168 Attribute Precedence ..........................................................................................................................171 Semantic Versioning Overview ...........................................................................................................171 Summary ......................................................................................................................172 ■ Chapter 8: Pulling It All Together: Continuous Provisioning with Chef and Azure ..................................................................................................173 What are we Aiming for? ..............................................................................................173 Phase 1 - Initial Chef Repository Setup ........................................................................175 Initializing the Application Repository ................................................................................................175 Add Chef Provisioning Recipes ...........................................................................................................176 Provisioning the CI Server ..................................................................................................................182 Phase 2 - Installing and Confi guring Jenkins ...............................................................185 Using Berkshelf to Retrieve Public Cookbooks ...................................................................................187 Creating a Recipe to Install Jenkins ...................................................................................................188 Uploading the Cookbook and Dependencies ......................................................................................189 viii ■ CONTENTS Preparing and Uploading Role Defi nitions ..........................................................................................189 Confi guring the Jenkins Server ..........................................................................................................190 Adding Plug-ins to Jenkins .................................................................................................................192 Verifying Jenkins Plug-in Installation .................................................................................................194 Securing Access to Jenkins ................................................................................................................195 Adding Chef Dependencies to Jenkins ...............................................................................................197 Phase 3 - Setting Up a New Chef Repository in GitHub ...............................................199 Confi guring a .gitignore File to Exclude Sensitive Information ...........................................................200 Connecting a Local Git Repo to the Remote .......................................................................................201 Phase 4 - Confi guring the Jenkins Project ...................................................................202 Adding a New Project .........................................................................................................................202 Adding Build Steps .............................................................................................................................204 Triggering a Jenkins Build from GitHub ..............................................................................................215 Adding and Destroying a Test Environment ........................................................................................217 Summary ......................................................................................................................219 ■ Appendix A: Further Resources .........................................................................221 Chef Server on the Azure Marketplace .........................................................................221 Azure Weekly Newsletter .............................................................................................222 Microsoft Azure Cookbook............................................................................................223 Conclusion ....................................................................................................................224 Index .....................................................................................................................225 ix