Python 4: Advanced Python Lesson 1: Going Further with Functions About Eclipse Perspectives and the Red Leaf Icon Working Sets Functions Are Objects Function Attributes Function and Method Calls Function Composition Lambdas: Anonymous Functions Quiz 1 Project 1 Lesson 2: Data Structures Organizing Data Handling Multi-Dimensional Arrays in Python Creating a Two-Dimensional Array List of Lists Example Using a Single List to Represent an Array Using an array.array instead of a List Using a dict instead of a List Summary Quiz 1 Project 1 Lesson 3: Delegation and Composition Extending Functionality by Inheritance More Complex Delegation Extending Functionality by Composition Recursive Composition Quiz 1 Project 1 Lesson 4: Publish and Subscribe On Program Structure Publish and Subscribe Publish and Subscribe in Action Validating Requests and Identifying Output Making the Algorithm More General A Note on Debugging Quiz 1 Project 1 Lesson 5: Optimizing Your Code Start with Correctness Where to Optimize The Profile Module Two Different Modules Using the Profile Module More Complex Reporting What to Optimize Loop Optimizations Pre-computing Attribute References Local Variables are Faster than Global Variables How to Optimize Don't Optimize Prematurely Use Timings, Not Intuition Make One Change at a Time The Best Way is Not Always Obvious Quiz 1 Project 1 Lesson 6: Using Exceptions Wisely Exceptions Are Not (Necessarily) Errors Specifying Exceptions Creating Exceptions and Raising Instances Using Exceptions Wisely Exception Timings Quiz 1 Project 1 Lesson 7: Advanced Uses of Decorators Decorator Syntax Classes as Decorators Class Decorators Odd Decorator Tricks Static and Class Method Decorators Parameterizing Decorators Quiz 1 Project 1 Lesson 8: Advanced Generators What Generators Represent Uses of Infinite Sequences The Itertools Module itertools.tee: duplicating generators itertools.chain() and itertools.islice(): Concatenating Sequences and Slicing Generators Like Lists itertools.count(), itertools.cycle() and itertools.repeat() itertools.dropwhile() and itertools.takewhile() Generator Expressions Quiz 1 Project 1 Lesson 9: Uses of Introspection The Meaning of 'Introspection' Some Simple Introspection Examples Attribute Handling Functions What Use is Introspection? The Inspect module The getmembers() Function Introspecting Functions Quiz 1 Project 1 Lesson 10: Multi-Threading Threads and Processes Multiprogramming Multiprocessing Multi-Threading Threading, Multiprocessing, CPython and the GIL The Threading Library Module Creating Threads (1) Waiting for Threads Creating Threads (2) Quiz 1 Project 1 Lesson 11: More on Multi-Threading Thread Synchronization threading.Lock Objects The Queue Standard Library Adding Items to Queues: Queue.put() Removing Items from Queues: Queue.get() Monitoring Completion: Queue.task_done() and Queue.join() A Simple Scalable Multi-Threaded Workhorse The Output Thread The Worker Threads The Control Thread Other Approaches Quiz 1 Project 1 Lesson 12: Multi-Processing The Multiprocessing Library Module multiprocessing Objects A Simple Multiprocessing Example A Multiprocessing Worker Process Pool The Output Process The Worker Process The Control Process Quiz 1 Project 1 Lesson 13: Functions and Other Objects A Deeper Look at Functions Required Keyword Arguments Function Annotations Nested Functions and Namespaces Partial Functions More Magic Methods How Python Expressions Work Quiz 1 Project 1 Lesson 14: Context Managers Another Python Control Structure: The With Statement Using a Simple Context Manager The Context Manager Protocol: __enter__() and __exit__() Writing Context Manager Classes Library Support for Context Managers Nested Context Managers Decimal Arithmetic and Arithmetic Contexts Decimal Arithmetic Contexts Decimal Signals The Default Decimal Context Quiz 1 Project 1 Lesson 15: Memory-Mapped Files Memory Mapping Memory-Mapped Files Are Still Files The mmap Interface What Use is mmap(), and How Does it Work? A Memory-Mapped Example Quiz 1 Project 1 Lesson 16: Your Future with Python Python Conferences Tutorials Talks The Hallway Track Open Space Lightning Talks Birds of a Feather Sessions (BOFs) Sprints: Moving Ahead The Python Job Market and Career Choices Python Development Tips and Tricks Quiz 1 Project 1 Copyright © 1998-2014 O'Reilly Media, Inc. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License. See http://creativecommons.org/licenses/by-sa/3.0/legalcode for more information. Going Further with Functions Welcome to the O'Reilly School of Technology (OST) Advanced Python course! We're happy you've chosen to learn Python programming with us. By the time you finish this course, you will have expanded your knowledge of Python and applied it to some really interesting technologies. Course Objectives When you complete this course, you will be able to: extend Python code functionality through inheritance, complex delegation, and recursive composition. publish, subscribe, and optimize your code. create advanced class decorators and generators in Python. demonstrate knowledge of Python introspection. apply multi-threading and mult-processing to Python development. manage arithmetic contexts and memory mapping. demonstrate understanding of the Python community, conferences, and job market. develop a multi-processing solution to a significant data processing problem. This course builds on your existing Python knowledge, incorporating further object-oriented design principles and techniques with the intention of rounding out your skill set. Techniques like recursion, composition, and delegation are explained and put into practice through the ever-present test-driven practical work. Learning with O'Reilly School of Technology Courses As with every O'Reilly School of Technology course, we'll take a user-active approach to learning. This means that you (the user) will be active! You'll learn by doing, building live programs, testing them and experimenting with them— hands-on! To learn a new skill or technology, you have to experiment. The more you experiment, the more you learn. Our system is designed to maximize experimentation and help you learn to learn a new skill. We'll program as much as possible to be sure that the principles sink in and stay with you. Each time we discuss a new concept, you'll put it into code and see what YOU can do with it. On occasion we'll even give you code that doesn't work, so you can see common mistakes and how to recover from them. Making mistakes is actually another good way to learn. Above all, we want to help you to learn to learn. We give you the tools to take control of your own learning experience. When you complete an OST course, you know the subject matter, and you know how to expand your knowledge, so you can handle changes like software and operating system updates. Here are some tips for using O'Reilly School of Technology courses effectively: Type the code. Resist the temptation to cut and paste the example code we give you. Typing the code actually gives you a feel for the programming task. Then play around with the examples to find out what else you can make them do, and to check your understanding. It's highly unlikely you'll break anything by experimentation. If you do break something, that's an indication to us that we need to improve our system! Take your time. Learning takes time. Rushing can have negative effects on your progress. Slow down and let your brain absorb the new information thoroughly. Taking your time helps to maintain a relaxed, positive approach. It also gives you the chance to try new things and learn more than you otherwise would if you blew through all of the coursework too quickly. Experiment. Wander from the path often and explore the possibilities. We can't anticipate all of your questions and ideas, so it's up to you to experiment and create on your own. Your instructor will help if you go completely off the rails. Accept guidance, but don't depend on it. Try to solve problems on your own. Going from misunderstanding to understanding is the best way to acquire a new skill. Part of what you're learning is problem solving. Of course, you can always contact your instructor for hints when you need them. Use all available resources! In real-life problem-solving, you aren't bound by false limitations; in OST courses, you are free to use any resources at your disposal to solve problems you encounter: the Internet, reference books, and online help are all fair game. Have fun! Relax, keep practicing, and don't be afraid to make mistakes! Your instructor will keep you at it until you've mastered the skill. We want you to get that satisfied, "I'm so cool! I did it!" feeling. And you'll have some projects to show off when you're done. Lesson Format We'll try out lots of examples in each lesson. We'll have you write code, look at code, and edit existing code. The code will be presented in boxes that will indicate what needs to be done to the code inside. Whenever you see white boxes like the one below, you'll type the contents into the editor window to try the example yourself. The CODE TO TYPE bar on top of the white box contains directions for you to follow: CODE TO TYPE: White boxes like this contain code for you to try out (type into a file to run). If you have already written some of the code, new code for you to add looks like this. If we want you to remove existing code, the code to remove will look like this. We may also include instructive comments that you don't need to type. We may run programs and do some other activities in a terminal session in the operating system or other command- line environment. These will be shown like this: INTERACTIVE SESSION: The plain black text that we present in these INTERACTIVE boxes is provided by the system (not for you to type). The commands we want you to type look lik e this. Code and information presented in a gray OBSERVE box is for you to inspect and absorb. This information is often color-coded, and followed by text explaining the code in detail: OBSERVE: Gray "Observe" boxes like this contain information (usually code specifics) for you to observe. The paragraph(s) that follow may provide addition details on information that was highlighted in the Observe box. We'll also set especially pertinent information apart in "Note" boxes: Note Notes provide information that is useful, but not absolutely necessary for performing the tasks at hand. Tip Tips provide information that might help make the tools easier for you to use, such as shortcut keys. WARNING Warnings provide information that can help prevent program crashes and data loss. Before you start programming in Python, let's review a couple of the tools you'll be using. If you've already taken the OST course on Introduction to Python, Getting More Out of Python and/or The Python Environment, you can skip to the next section if you like, or you might want to go through this section to refresh your memory. About Eclipse We use an Integrated Development Environment (IDE) called Eclipse. It's the program filling up your screen right now. IDEs assist programmers by performing tasks that need to be done repetitively. IDEs can also help to edit and debug code, and organize projects. Perspectives and the Red Leaf Icon The Ellipse Plug-in for Eclipse was developed by OST. It adds a Red Leaf icon to the toolbar in Eclipse. This icon is your "panic button." Because Eclipse is versatile and allows you to move things around, like views, toolbars, and such, it's possible to lose your way. If you do get confused and want to return to the default perspective (window layout), the Red Leaf icon is the fastest and easiest way to do that. To use the Red Leaf icon to: reset the current perspective: click the icon. change perspectives: click the drop-down arrow beside the icon to select a perspective. select a perspective: click the drop-down arrow beside the Red Leaf icon and select the course (Java, Python, C++, etc.). Selecting a specific course opens the perspective designed for that particular course. For this course, select Python: Working Sets In this course, we'll use working sets. All projects created in Eclipse exist in the workspace directory of your account on our server. As you create projects throughout the course, your directory could become pretty cluttered. A working set is a view of the workspace that behaves like a folder, but it's actually an association of files. Working sets allow you to limit the detail that you see at any given time. The difference between a working set and a folder is that a working set doesn't actually exist in the file system. A working set is a convenient way to group related items together. You can assign a project to one or more working sets. In some cases, like the Python extension to Eclipse, new projects are created in a catch-all "Other Projects" working set. To organize your work better, we'll have you assign your projects to an appropriate working set when you create them. To do that, you'l right-click on the project name and select the Assign Working Sets menu item. We've already created some working sets for you in the Eclipse IDE. You can turn the working set display on or off in Eclipse. For this course, we'll display only the working sets you need. In the upper-right corner of the Package Explorer panel, click the downward arrow and select Configure Working Sets: Select the Other Projects working set as well as the ones that begin with "Python4," then click OK: Let's create a project to store our programs for this lesson. Select File | New | Pydev Project, and enter the information as shown: Click Finish. When asked if you want to open the associated perspective, check the Remember my decision box and click No: By default, the new project is added to the Other Projects working set. Find Python4_Lesson01 there, right- click it, and select Assign Working Sets... as shown: Select the Python4_Lessons working set and click OK: