C 3 o r ve d r with s PytEdit 3. h i 6 o o hi n n ghli2.7 ghts& 3. 5 Python in a Nutshell THE DEFINITIVE REFERENCE Alex Martelli, Anna Ravenscroft & Steve Holden PYTHON IN A NUTSHELL Third Edition Alex Martelli, Anna Ravenscroft, and Steve Holden Python in a Nutshell by Alex Martelli, Anna Ravenscroft, and Steve Holden Copyright © 2017 Alex Martelli, Anna Martelli Ravenscroft, and Steve Holden. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com/safari). For more information, con‐ tact our corporate/institutional sales department: 800-998-9938 or [email protected]. Editor: Dawn Schanafelt Indexer: Judith McConville Production Editor: Kristen Brown Interior Designer: David Futato Copyeditor: Kim Cofer Cover Designer: Karen Montgomery Proofreader: Rachel Monaghan Illustrator: Rebecca Demarest April 2017: Third Edition Revision History for the Third Edition 2017-04-06: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781449392925 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Python in a Nutshell, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. While the publisher and the authors have used good faith efforts to ensure that the informa‐ tion and instructions contained in this work are accurate, the publisher and the authors dis‐ claim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technol‐ ogy this work contains or describes is subject to open source licenses or the intellectual prop‐ erty rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 978-1-449-39292-5 [LSI] Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Part I. Getting Started with Python 1. Introduction to Python. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 The Python Language 3 The Python Standard Library and Extension Modules 5 Python Implementations 5 Python Development and Versions 12 Python Resources 14 Installation 17 Installing Python from Binaries 18 Installing Python from Source Code 19 Installing Jython 21 Installing IronPython 22 Installing PyPy 22 2. The Python Interpreter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 The python Program 23 Python Development Environments 28 Running Python Programs 30 The jython Interpreter 31 The IronPython Interpreter 31 The PyPy Interpreter 32 iii Part II. Core Python Language and Built-ins 3. The Python Language. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Lexical Structure 35 Data Types 41 Variables and Other References 50 Expressions and Operators 55 Numeric Operations 57 Sequence Operations 59 Set Operations 65 Dictionary Operations 67 Control Flow Statements 70 Functions 80 4. Object-Oriented Python. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Classes and Instances 99 Special Methods 123 Decorators 138 Metaclasses 140 5. Exceptions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 The try Statement 147 The with Statement and Context Managers 151 Exception Propagation 153 The raise Statement 155 Exception Objects 156 Custom Exception Classes 161 Error-Checking Strategies 163 The assert Statement 168 6. Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 Module Objects 172 Module Loading 176 Packages 182 Distribution Utilities (distutils) and setuptools 184 Python Environments 186 7. Core Built-ins and Standard Library Modules. . . . . . . . . . . . . . . . . . . . . . . . . 195 Built-in Types 196 Built-in Functions 199 The sys Module 210 iv | Table of Contents The copy Module 214 The collections Module 215 The functools Module 220 The heapq Module 222 The argparse Module 225 The itertools Module 226 8. Strings and Things. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 Methods of String and Bytes Objects 231 The string Module 237 String Formatting 238 Text Wrapping and Filling 247 The pprint Module 247 The reprlib Module 248 Unicode 248 9. Regular Expressions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251 Regular Expressions and the re Module 251 Part III. Python Library and Extension Modules 10. File and Text Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 Other Chapters That Also Deal with Files 269 Organization of This Chapter 269 The io Module 271 Auxiliary Modules for File I/O 279 In-Memory “Files”: io.StringIO and io.BytesIO 284 Compressed Files 285 The os Module 295 Filesystem Operations 297 Text Input and Output 315 Richer-Text I/O 317 Interactive Command Sessions 323 Internationalization 327 11. Persistence and Databases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 Serialization 336 DBM Modules 346 Berkeley DB Interfacing 349 The Python Database API (DBAPI) 2.0 349 Table of Contents | v 12. Time Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361 The time Module 361 The datetime Module 366 The pytz Module 372 The dateutil Module 373 The sched Module 375 The calendar Module 377 13. Controlling Execution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379 Site and User Customization 379 Termination Functions 381 Dynamic Execution and exec 381 Internal Types 385 Garbage Collection 386 14. Threads and Processes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393 Threads in Python 394 The threading Module 395 The queue Module 404 The multiprocessing Module 406 The concurrent.futures Module 415 Threaded Program Architecture 418 Process Environment 421 Running Other Programs 422 The mmap Module 427 15. Numeric Processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431 The math and cmath Modules 431 The operator Module 436 Random and Pseudorandom Numbers 439 The fractions Module 441 The decimal Module 442 The gmpy2 Module 443 Array Processing 444 The array Module 444 Extensions for Numeric Array Computation 446 16. Testing, Debugging, and Optimizing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453 Testing 454 Debugging 465 The warnings Module 474 Optimization 477 vi | Table of Contents Part IV. Network and Web Programming 17. Networking Basics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495 Networking Principles 496 The Berkeley Socket Interface 497 Transport Layer Security (TLS, AKA SSL) 512 18. Asynchronous Alternatives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515 Coroutine-Based Async Architectures 516 The asyncio Module (v3 Only) 517 The selectors Module 540 19. Client-Side Network Protocol Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547 Email Protocols 548 HTTP and URL Clients 551 Other Network Protocols 565 20. Serving HTTP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567 WSGI 568 Python Web Frameworks 569 21. Email, MIME, and Other Network Encodings. . . . . . . . . . . . . . . . . . . . . . . . . . 591 MIME and Email Format Handling 591 Encoding Binary Data as ASCII Text 601 22. Structured Text: HTML. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 605 The html.entities (v2: htmlentitydefs) Module 606 Generating HTML 617 23. Structured Text: XML. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625 ElementTree 626 Part V. Extending, Distributing, v2/v3 Migration 24. Extending and Embedding Classic Python. . . . . . . . . . . . . . . . . . . . . . . . . . . . 641 Extending Python with Python’s C API 642 Extending Python Without Python’s C API 677 Cython 678 Embedding Python 683 Table of Contents | vii 25. Distributing Extensions and Programs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 687 setuptools 688 Distributing Your Package 698 26. v2/v3 Migration and Coexistence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 705 Preparing for Python 3 706 Minimizing Syntax Differences 707 Choosing Your Support Strategy 711 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 723 viii | Table of Contents