CYAN YELLOW MAGENTA BLACK PANTONE 123 C BOOKS FOR PROFESSIONALS BY PROFESSIONALS® THE EXPERT’S VOICE® IN OPEN SOURCE Companion eBook Available Pro Python System Administration Pro Dear Reader, P Pro Python System Administration takes you beyond standard management tools and quick-and-dirty shell scripts. You’ll discover real-world examples of y customized tools built using Python and the Django and Jinja2 frameworks – t tools that you can put to work in your own environment right away. h Each chapter explains how to determine your requirements, how to create Pro Rytis Sileika your tools, and how to use them effectively. I chose the Python programming o language because it's relatively simple to use, easy to learn, and elegant. It also n comes with a large base of additional modules. Python S These are some of the practical solutions you’ll learn to create: • How to manage network devices with SOAP and SNMP y • How to monitor web servers and databases using Django and Jinja2 s • How to perform statistical computation with NumPy • How to offload processing tasks to Amazon Web Services t e Administering network systems, a complex task in itself, requires a variety of m specialized skills. This book will help you build the customized solutions you need to create an efficient, productive environment. A System Administration Rytis Sileika, RHCE d m THE APRESS ROADMAP Beginning Python, i Second Edition Beginning n Pro Python Python Visualization Python 3 for i Companion eBook Absolute Beginners s Learn to manage and monitor your network, Foundations Pro t web servers, and databases with Python of Agile Python Python System r Dive into Development Administration Python 3 a t See last page for details i on $10 eBook version o n Rytis Sileika SOURCE CODE ONLINE S www.apress.com ISBN 978-1-4302-2605-5 il e 54999 i k a Shelve in: Python User level: 9 781430 226055 Intermediate–Advanced this print for content only—size & color not accurate 7.5 x 9.25 spine = 0.78125" 416 page count Pro Python System Administration ■ ■ ■ Rytis Sileika Pro Python System Administration Copyright © 2010 by Rytis Sileika All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher. ISBN-13 (pbk): 978-1-4302-2605-5 ISBN-13 (electronic): 978-1-4302-2606-2 Printed and bound in the United States of America 9 8 7 6 5 4 3 2 1 Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. President and Publisher: Paul Manning Lead Editors: Duncan Parkes and Michelle Lowman Technical Reviewer: Patrick Engebretson Editorial Board: Clay Andres, Steve Anglin, Mark Beckner, Ewan Buckingham, Gary Cornell, Jonathan Gennick, Jonathan Hassell, Michelle Lowman, Matthew Moodie, Duncan Parkes, Jeffrey Pepper, Frank Pohlmann, Douglas Pundick, Ben Renow-Clarke, Dominic Shakeshaft, Matt Wade, Tom Welsh Coordinating Editors: Mary Tobin and Jennifer L. Blackwell Copy Editors: Jim Compton, Heather Lang and Marilyn Smith Compositor: Lynn L’Heureux Indexer: Julie Grady Artist: April Milne Cover Designer: Anna Ishchenko Distributed to the book trade worldwide by Springer Science+Business Media, LLC., 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail [email protected], or visit www.springeronline.com. For information on translations, please e-mail [email protected], or visit www.apress.com. Apress and friends of ED books may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Special Bulk Sales–eBook Licensing web page at www.apress.com/info/bulksales. The information in this book is distributed on an “as is” basis, without warranty. Although every precaution has been taken in the preparation of this work, neither the author(s) nor Apress shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in this work. The source code for this book is available to readers at www.apress.com. I want to dedicate this book to my family—my wife Evelina and daughters Gabija and Milda ■CONTENTS Contents at a Glance ■About the Author.................................................................................................................xvi ■About the Technical Reviewer...........................................................................................xvii ■Acknowledgements..........................................................................................................xviii ■Introduction.........................................................................................................................xix ■Chapter 1: Reading and Collecting Performance Data Using SNMP....................................1 ■Chapter 2: Managing Devices Using the SOAP API.............................................................41 ■Chapter 3: Creating a Web Application for IP Address Accountancy................................75 ■Chapter 4: Integrating the IP Address Application with DHCP.........................................107 ■Chapter 5: Maintaining a List of Virtual Hosts in an Apache Configuration File.............137 ■Chapter 6: Gathering and Presenting Statistical Data from Apache Log Files....................159 ■Chapter 7: Performing Complex Searches and Reporting on Application Log Files.......181 ■Chapter 8: A Web Site Availability Check Script for Nagios.............................................211 ■Chapter 9: Management and Monitoring Subsystem.......................................................231 ■Chapter 10: Remote Monitoring Agents............................................................................269 ■Chapter 11: Statistics Gathering and Reporting...............................................................297 ■Chapter 12: Automatic MySQL Database Performance Tuning........................................329 ■Chapter 13: Using Amazon EC2/S3 as a Data Warehouse Solution.................................349 ■Index...................................................................................................................................373 v ■CONTENTS Contents ■About the Author.................................................................................................................xvi ■About the Technical Reviewer...........................................................................................xvii ■Acknowledgements..........................................................................................................xviii ■Introduction.........................................................................................................................xix ■Chapter 1: Reading and Collecting Performance Data Using SNMP....................................1 Application Requirements and Design......................................................................................1 Specifying the Requirements.............................................................................................1 High-Level Design Specification.........................................................................................2 Introduction to SNMP...............................................................................................................2 The System SNMP Variables Node.....................................................................................4 The Interfaces SNMP Variables Node.................................................................................5 Authentication in SNMP.....................................................................................................7 Querying SNMP from the Command Line...........................................................................7 Querying SNMP Devices from Python.....................................................................................11 Configuring the Application..............................................................................................11 Using the PySNMP Library...............................................................................................13 Implementing the SNMP Read Functionality.....................................................................18 Storing Data with RRDTool.....................................................................................................18 Introduction to RRDTool...................................................................................................18 Using RRDTool from a Python Program............................................................................20 Creating a RoundRobin Database.....................................................................................20 Writing and Reading Data from the RoundRobin Database...............................................22 Plotting Graphs with RRDTool..........................................................................................25 Integrating RRDTool with the Monitoring Solution.............................................................28 Creating Web Pages with the Jinja2 Templating System.........................................................31 Loading Template Files with Jinja2..................................................................................32 The Jinja2 Template Language........................................................................................33 Generating Web Site Pages..............................................................................................35 Summary...............................................................................................................................39 vii ■CONTENTS ■Chapter 2: Managing Devices Using the SOAP API.............................................................41 What Is the SOAP API?............................................................................................................41 The Structure of a SOAP Message....................................................................................42 Requesting Services with SOAP.......................................................................................42 Finding Information about Available Services with WSDL..................................................44 SOAP Support in Python.........................................................................................................45 Converting WSDL Schema to Python Helper Module...............................................................46 Defining Requirements for Our Load Balancer Tool.................................................................47 Basic Requirements.........................................................................................................48 Code Structure.................................................................................................................48 Configuration...................................................................................................................49 Accessing Citrix Netscaler Load Balancer with the SOAP API..................................................50 Fixing Issues with Citrix Netscaler WSDL..........................................................................50 Creating a Connection Object...........................................................................................51 Logging In: Our First SOAP Call........................................................................................54 Gathering Performance Statistics Data...................................................................................59 SOAP Methods for Reading Statistical Data and Their Return Values................................59 Reading System Health Data............................................................................................60 Reading Service Status Data............................................................................................62 Automating Administration Tasks...........................................................................................65 Device Configuration SOAP Methods................................................................................66 Setting a Service State.....................................................................................................66 A Word About Logging and Error Handling..............................................................................68 Using the Python logging Module..................................................................................68 Handling Exceptions........................................................................................................72 Summary...............................................................................................................................73 ■Chapter 3: Creating a Web Application for IP Address Accountancy................................75 Designing the Application.......................................................................................................75 Setting Out the Requirements..........................................................................................75 Making Design Decisions.................................................................................................76 Defining the Database Schema........................................................................................77 Creating the Application Workflow...................................................................................79 viii