SECOND EDITION Twisted Network Programming Essentials Jessica McKellar and Abe Fettig Twisted Network Programming Essentials, Second Edition by Jessica McKellar and Abe Fettig Copyright © 2013 Jessica McKellar. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://my.safaribooksonline.com). For more information, contact our corporate/ institutional sales department: 800-998-9938 or [email protected]. Editor: Meghan Blanchette Indexer: Bob Pfahler Production Editor: Christopher Hearse Cover Designer: Randy Comer Copyeditor: Rachel Head Interior Designer: David Futato Proofreader: Amanda Kersey Illustrator: Rebecca Demarest March 2013: Second Edition Revision History for the Second Edition: 2013-03-11: First release See http://oreilly.com/catalog/errata.csp?isbn=9781449326111 for release details. Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc. Twisted Network Programming Essentials, 2nd Edition, an image of a ball of snakes, and related trade dress are trademarks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc., was aware of a trade‐ mark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein. ISBN: 978-1-449-32611-1 [LSI] Table of Contents Foreword to the First Edition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Part I. An Introduction to Twisted 1. Getting Started. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Installing Twisted 3 Installation on Linux 3 Installation on Windows 4 Installation on OS X 5 Installing from Source 5 Required Dependencies 5 Installing Twisted from a Release Tarball 6 Installing Twisted from a Source Checkout 6 Installing Optional Dependencies from Source 6 Testing Your Installation 7 Using the Twisted Documentation 8 API Documentation 8 Subproject Documentation 8 Finding Answers to Your Questions 8 Mailing Lists 9 IRC Channels 9 Stack Overflow 9 Twisted Blogs 9 2. Building Basic Clients and Servers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 A TCP Echo Server and Client 11 Event-Driven Programming 12 The Reactor 14 iii Transports 14 Protocols 15 Protocol Factories 16 Decoupling Transports and Protocols 16 A TCP Quote Server and Client 16 Protocol State Machines 19 More Practice and Next Steps 22 3. Writing Asynchronous Code with Deferreds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 What Deferreds Do and Don’t Do 25 The Structure of a Deferred Object 26 Callback Chains and Using Deferreds in the Reactor 28 Practice: What Do These Deferred Chains Do? 30 Exercise 1 31 Exercise 2 31 Exercise 3 31 Exercise 4 32 Exercise 5 32 Exercise 6 33 The Truth About addCallbacks 33 Exercise 7 34 Exercise 8 35 Key Facts About Deferreds 35 Summary of the Deferred API 36 More Practice and Next Steps 36 4. Web Servers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Responding to HTTP Requests: A Low-Level Review 39 The Structure of an HTTP Request 40 Parsing HTTP Requests 42 Handling GET Requests 43 Serving Static Content 43 Serving Dynamic Content 45 Dynamic Dispatch 46 Handling POST Requests 48 A Minimal POST Example 48 Asynchronous Responses 49 More Practice and Next Steps 51 5. Web Clients. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Basic HTTP Resource Retrieval 53 Printing a Web Resource 53 iv | Table of Contents Downloading a Web Resource 54 Agent 55 Requesting Resources with Agent 55 Retrieving Response Metadata 57 POSTing Data with Agent 58 More Practice and Next Steps 60 Part II. Building Production-Grade Twisted Services 6. Deploying Twisted Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 The Twisted Application Infrastructure 63 Services 64 Applications 64 TAC Files 64 twistd 65 Plugins 66 More twistd Examples 68 More Practice and Next Steps 68 Suggested Exercises 69 7. Logging. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 Basic In-Application Logging 71 twistd Logging 73 Custom Loggers 73 Key Facts and Caveats About Logging 75 8. Databases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 Nonblocking Database Queries 77 More Practice and Next Steps 80 9. Authentication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 The Components of Twisted Cred 81 Twisted Cred: An Example 82 Credentials Checkers 86 Authentication in Twisted Applications 89 More Practice and Next Steps 91 10. Threads and Subprocesses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 Threads 93 Subprocesses 96 Running a Subprocess and Getting the Result 96 Table of Contents | v Custom Process Protocols 97 More Practice and Next Steps 100 11. Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Writing and Running Twisted Unit Tests with Trial 103 Testing Protocols 104 Tests and the Reactor 108 Testing Deferreds 109 Testing the Passage of Time 112 More Practice and Next Steps 115 Part III. More Protocols and More Practice 12. Twisted Words. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 IRC Clients 119 IRC Servers 121 More Practice and Next Steps 124 13. Twisted Mail. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 SMTP Clients and Servers 126 The SMTP Protocol 126 Sending Emails Using SMTP 127 SMTP Servers 128 Storing Mail 130 IMAP Clients and Servers 132 IMAP Servers 133 IMAP Clients 137 POP3 Clients and Servers 139 POP3 Servers 139 More Practice and Next Steps 143 14. SSH. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 SSH Servers 145 A Basic SSH Server 145 Using Public Keys for Authentication 151 Providing an Administrative Python Shell 153 Running Commands on a Remote Server 156 SSH Clients 156 More Practice and Next Steps 159 15. The End. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 vi | Table of Contents Contributing to Twisted 161 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 Table of Contents | vii