Automating Linux and Unix System Administration Second Edition Nate Campi and Kirk Bauer Automating Linux and Unix System Administration, Second Edition Copyright © 2009 by Nate Campi, Kirk Bauer All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher. ISBN-13 (pbk): 978-1-4302-1059-7 ISBN-13 (electronic): 978-1-4302-1060-3 Printed and bound in the United States of America 9 8 7 6 5 4 3 2 1 Trademarked names may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, we use the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. Lead Editor: Frank Pohlmann Technical Reviewer: Mark Burgess Editorial Board: Clay Andres, Steve Anglin, Mark Beckner, Ewan Buckingham, Tony Campbell, Gary Cor- nell, Jonathan Gennick, Michelle Lowman, Matthew Moodie, Jeffrey Pepper, Frank Pohlmann, Ben Renow-Clarke, Dominic Shakeshaft, Matt Wade, Tom Welsh Project Manager: Kylie Johnston Copy Editors: Nina Goldschlager, Heather Lang Associate Production Director: Kari Brooks-Copony Production Editor: Ellie Fountain Compositor: Linda Weidemann, Wolf Creek Press Proofreader: Nancy Sixsmith Indexer: Becky Hornyak Cover Designer: Kurt Krames Manufacturing Director: Tom Debolski Distributed to the book trade worldwide by Springer-Verlag New York, Inc., 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax 201-348-4505, e-mail (cid:107)(cid:110)(cid:96)(cid:97)(cid:110)(cid:111)(cid:41)(cid:106)(cid:117)(cid:60)(cid:111)(cid:108)(cid:110)(cid:101)(cid:106)(cid:99)(cid:97)(cid:110)(cid:41)(cid:111)(cid:94)(cid:105)(cid:42)(cid:95)(cid:107)(cid:105), or visit (cid:100)(cid:112)(cid:112)(cid:108)(cid:54)(cid:43)(cid:43)(cid:115)(cid:115)(cid:115)(cid:42)(cid:111)(cid:108)(cid:110)(cid:101)(cid:106)(cid:99)(cid:97)(cid:110)(cid:107)(cid:106)(cid:104)(cid:101)(cid:106)(cid:97)(cid:42)(cid:95)(cid:107)(cid:105). For information on translations, please contact Apress directly at 2855 Telegraph Avenue, Suite 600, Berkeley, CA 94705. Phone 510-549-5930, fax 510-549-5939, e-mail (cid:101)(cid:106)(cid:98)(cid:107)(cid:60)(cid:93)(cid:108)(cid:110)(cid:97)(cid:111)(cid:111)(cid:42)(cid:95)(cid:107)(cid:105), or visit (cid:100)(cid:112)(cid:112)(cid:108)(cid:54)(cid:43)(cid:43)(cid:115)(cid:115)(cid:115)(cid:42)(cid:93)(cid:108)(cid:110)(cid:97)(cid:111)(cid:111)(cid:42)(cid:95)(cid:107)(cid:105). Apress and friends of ED books may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Special Bulk Sales–eBook Licensing web page at (cid:100)(cid:112)(cid:112)(cid:108)(cid:54)(cid:43)(cid:43)(cid:115)(cid:115)(cid:115)(cid:42)(cid:93)(cid:108)(cid:110)(cid:97)(cid:111)(cid:111)(cid:42)(cid:95)(cid:107)(cid:105)(cid:43)(cid:101)(cid:106)(cid:98)(cid:107)(cid:43)(cid:94)(cid:113)(cid:104)(cid:103)(cid:111)(cid:93)(cid:104)(cid:97)(cid:111). The information in this book is distributed on an “as is” basis, without warranty. Although every pre- caution has been taken in the preparation of this work, neither the author(s) nor Apress shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in this work. The source code for this book is available to readers at (cid:100)(cid:112)(cid:112)(cid:108)(cid:54)(cid:43)(cid:43)(cid:115)(cid:115)(cid:115)(cid:42)(cid:93)(cid:108)(cid:110)(cid:97)(cid:111)(cid:111)(cid:42)(cid:95)(cid:107)(cid:105). I dedicate this book to my dear grandmother Mary Lou. Her influence makes everyone around her a better person, and her presence lights up a room. She is beautiful inside and out, and she meets adversity with faith, quiet dignity, and grace. —Nate Campi Contents at a Glance About the Authors ................................................................. xv About the Technical Reviewer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii Acknowledgments ................................................................xix Introduction .....................................................................xxi CHAPTER 1 Introducing the Basics of Automation ............................1 CHAPTER 2 Applying Practical Automation ..................................19 CHAPTER 3 Using SSH to Automate System Administration Securely .........27 CHAPTER 4 Configuring Systems with cfengine .............................49 CHAPTER 5 Bootstrapping a New Infrastructure .............................79 CHAPTER 6 Setting Up Automated Installation ..............................107 CHAPTER 7 Automating a New System Infrastructure .......................161 CHAPTER 8 Deploying Your First Application ...............................213 CHAPTER 9 Generating Reports and Analyzing Logs ........................253 CHAPTER 10 Monitoring ....................................................273 CHAPTER 11 Infrastructure Enhancement ...................................323 CHAPTER 12 Improving System Security ....................................353 APPENDIX A Introducing the Basic Tools ....................................375 APPENDIX B Writing cfengine Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .395 ฀INDEX .......................................................................401 v Contents About the Authors ................................................................. xv About the Technical Reviewer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii Acknowledgments ................................................................xix Introduction .....................................................................xxi CHAPTER 1 Introducing the Basics of Automation ......................1 Do You Need Automation? .........................................2 Large Companies with Many Diverse Systems ...................4 Medium-Sized Companies Planning for Growth ..................4 Internet Service Providers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 Application Service Providers ..................................5 Web Server Farms ...........................................5 Beowulf Clusters ............................................6 Network Appliances ..........................................7 What Will You Gain? ..............................................7 Saving Time ................................................7 Reducing Errors .............................................7 Documenting System Configuration Policies ....................8 Realizing Other Benefits ......................................8 What Do System Administrators Do? ...............................10 Methodology: Get It Right from the Start! ...........................11 Homogenizing Your Systems .................................13 Deciding on Push vs. Pull ....................................13 Dealing with Users and Administrators .............................14 Who Owns the Systems? .........................................17 Defining Policy ..................................................18 vii viii (cid:78)CONTENTS CHAPTER 2 Applying Practical Automation .............................19 Seeing Everything As a File .......................................19 Understanding the Procedure Before Automating It ..................20 Exploring an Example Automation .................................21 Scripting a Working Procedure ...............................21 Prototyping Before You Polish ................................22 Turning the Script into a Robust Automation ...................23 Attempting to Repair, Then Failing Noisily ......................24 Focusing on Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25 CHAPTER 3 Using SSH to Automate System Administration Securely .....................................................27 Learning the Basics of Using SSH ..................................28 Enhancing Security with SSH .....................................29 Using Public- Key Authentication ...................................30 Generating the Key Pair .....................................31 Specifying Authorized Keys ..................................32 Using ssh- agent .................................................33 Knowing ssh- agent Basics ...................................33 Getting Advanced with s sh- agent .............................34 Forwarding Keys. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36 Restricting RSA Authentication ....................................37 Dealing with Untrusted Hosts .................................38 Allowing Limited Command Execution .........................38 Forwarding a Port ...........................................39 Using SSH for Common Accounts ..................................40 Preparing for Common Accounts .............................41 Monitoring the Common Accounts ............................45 (cid:78)CONTENTS ix CHAPTER 4 Configuring Systems with cfengine .......................49 Getting an Overview of cfengine ...................................49 Defining cfengine Concepts ..................................49 Evaluating Push vs. Pull .....................................51 Delving into the Components of cfengine ......................53 Mapping the cfengine Directory Structure ......................53 Managing cfengine Configuration Files ........................54 Identifying Systems with Classes .............................55 Finding More Information About Cfengine ......................57 Learning the Basic Setup .........................................58 Setting Up the Network ......................................58 Running Necessary Processes ................................58 Creating Basic Configuration Files ............................60 Creating the Configuration Server .............................64 Preparing the Client Systems .................................65 Debugging cfengine .............................................66 Creating Sections in cfagent.conf ..................................66 Using Classes in cfagent.conf ................................67 The copy Section ...........................................68 The directories Section ......................................69 The disable Section .........................................69 The editfiles Section ........................................71 The files Section ............................................72 The links Section ...........................................74 The processes Section ......................................74 The shellcommands Section .................................75 Using cfrun .....................................................75 Looking Forward to Cfengine 3 ....................................76 Using cfengine in the Real World ..................................77 CHAPTER 5 Bootstrapping a New Infrastructure .......................79 Installing the Central cfengine Host ................................80 Setting Up the cfengine Master Repository .........................81 x (cid:78)CONTENTS Creating the cfengine Config Files ................................82 The cf.preconf Script ........................................82 The update.conf file .........................................88 The cfagent.conf file ........................................92 The cf.motd Task ...........................................99 The cf.cfengine_cron_entries Task ..........................102 cfservd.conf ..............................................103 Ready for Action ................................................105 CHAPTER 6 Setting Up Automated Installation ........................107 Introducing the Example Environment .............................108 FAI for Debian ............................................109 Employing JumpStart for Solaris ............................122 Kickstart for Red Hat .......................................136 The Proper Foundation ..........................................158 CHAPTER 7 Automating a New System Infrastructure ................161 Implementing Time Synchronization ..............................161 External NTP Synchronization ...............................162 Internal NTP Masters .......................................163 Configuring the NTP Clients .................................164 Copying the Configuration Files with cfengine .................166 An Alternate Approach to Time Synchronization ...............170 Incorporating DNS ..............................................170 Choosing a DNS Architecture ................................171 Setting Up Private DNS .....................................171 Taking Control of User Account Files ..............................188 Standardizing the Local Account Files ........................188 Distributing the Files with cfengine ..........................191 Adding New User Accounts .................................196 Routing Mail ...................................................208 Looking Back ..................................................211 (cid:78)CONTENTS xi CHAPTER 8 Deploying Your First Application ..........................213 Deploying and Configuring the Apache Web Server .................213 The Apache Package from Red Hat ..........................213 Building Apache from Source ...............................216 Sharing Data Between Systems ..................................218 Synchronizing Data with rsync ..............................218 Sharing Data with NFS .....................................232 Sharing Program Binaries with NFS ..........................235 Sharing Data with cfengine .................................240 Sharing Data with Subversion ...............................242 NFS and rsync and cfengine, Oh My! ..............................251 CHAPTER 9 Generating Reports and Analyzing Logs .................253 Reporting on cfengine Status ....................................253 Doing General syslog Log Analysis ................................263 Configuring the syslog Server ...............................263 Outputting Summary Log Reports ...........................267 Doing R eal-T ime Log Reporting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .269 Seeing the Light ................................................272 CHAPTER 10 Monitoring ..................................................273 Nagios ........................................................274 Nagios Components ........................................275 Nagios Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .276 Deploying Nagios with cfengine .............................278 Create the Nagios Web Interface Configuration Files ...........284 NRPE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .297 Monitoring Remote Systems ................................306 What Nagios Alerts Really Mean .............................312 Ganglia ........................................................312 Building and Distributing the Ganglia Programs ................313 Configuring the Ganglia Web Interface ........................318 Now You Can Rest Easy .........................................321 xii (cid:78)CONTENTS CHAPTER 11 Infrastructure Enhancement ..............................323 Cfengine Version Control with Subversion .........................323 Importing the masterfiles Directory Tree ......................323 Using Subversion to Implement a Testing Environment .........331 Backups .......................................................337 Jumpstart ................................................338 Kickstart ..................................................340 FAI .......................................................342 Subversion Backups .......................................346 Enhancement Is an Understatement ..............................352 CHAPTER 12 Improving System Security ................................353 Security Enhancement with cfengine ..............................354 Removing the SUID Bit .....................................355 Protecting System Accounts ................................359 Applying Patches and Vendor Updates ........................360 Shutting Down Unneeded Daemons ..........................361 Removing Unsafe Files .....................................362 File Checksum Monitoring ..................................363 Using the Lightweight Directory Access Protocol ...................364 Security with Kerberos ..........................................365 Implementing Host-Based Firewalls ...............................365 Using TCP Wrappers .......................................366 Using Host-Based Packet Filtering ...........................367 Enabling Sudo at Our Example Site ...............................371 Security Is a Journey, Not a Destination ...........................374 APPENDIX A Introducing the Basic Tools ...............................375 The Bash Shell .................................................375 Compatibility Issues with Bash ..............................376 Creating Simple Bash Shell Scripts. . . . . . . . . . . . . . . . . . . . . . . . . . .376 Debugging Bash Scripts ....................................377 Other Shells ...............................................378 Bash Resources ...........................................379