Python 3 Combat Guide A comprehensive manual about tricks, good practices and coding style for hands on Software Developer Managers and Developers By Carles Mateo Creation: v. 0.1 2014-01-01 10:10 Catalonia Time Last Updated: v.0.99 2021-06-18 23:00 Ireland Time Python Combat Guide 1 / 365 Carles Mateo Dedication I want to dedicate this book to my mother, who raised two children alone, in a hostile and macho world. She is specially responsible of my success for believing in me, and for buying me a MSX computer when I was 5 years old, as I mentioned that I found my passion. Thanks I want to thank all my colleagues in MPSTOR (Sanmina’s Software Factory in Ireland for the Hardware division), the first company I worked for in Ireland, for making me feel like we were a family, and being always collaborative at the pair with being great Engineers. You’re real friends guys!. I want to thank specially to my former Manager and Engineering Director, Seán Whelan, from whom I learned so much and who shared with me so many interesting conversations the Fridays, in those coffees in Mr Cotton1, and which I considered so valuable that I use to say that was my real bonus (and not the bonus from my salary). Seán supported me with my DRAID project for rebuilding RAID drives very quickly, and really helped me also to deal with bureaucracy and to deal with all the diatribes or making a Software product that runs in the Hardware of a multinational company, so contacting the right Hardware departments in Colorado for reporting Firmware bugs I found, coordinating the Engineering Testing, reporting to CEO, conducting EVT2 across multiple Teams, etc… I was also a bit ashamed of my “robot Terminator” English, and he was the first person to tell me that my English was really good, only my accent was to be polished. After, many kind Irish people told me that I have a really beautiful exotic accent. I also want to thank to all the members of my Team in Sanmina and the ones that contributed to the DRAID project, too many to mention, but in special the ones I hired: • Kevin Maher, for creating MPMGR, the Python tool to manage DRAID and ZFS • Dlo Bagari, for bringing that extra, bringing in scientific tools, and making the C changes required to DRAID and ZFS code, in order to have the protection for the Data I wanted when a drive dies suddenly • Síle Neylon, for catching impossible bugs that nobody else will • Franklyn Aghedo, for conducting Engineering Tests in a very professional and automated way Also to my loving colleagues of Blizzard, which make me feel like home, and make me enjoy so much by sharing their knowledge with me and being real friends. And to my Manager, Bianca, who not only does not cut my wings, but encourages me to go 1 Mr Cotton is a local business in Cork, which delivers a small ban, the size of an Smart car, with an Italian professional coffee machine, and parks it in Business Parks at certain hours and sells coffee. 2 Engineering Validation Tests. Python Combat Guide 2 / 365 Carles Mateo for more, to reach higher limits, to solve the unsolvable, and to keep learning at my natural brutal speed, and also for offering brilliant advice, sharing arcane knowledge, and offering me his common sense too where my Engineer mind would go binary. I would not be fair if I don’t mention all my colleagues at GNOC and OpsCenter. We are a Super Team!. I don’t mention all the names for security concerns. I’ll only mention the ones that reviewed this book before becoming public: Michela, Sid, Gunnar, Miranda, Nathaniel, Thomas and my best friend Edu H. Thanks. Also a memory for my long-term Engineer friends: J.A. Garrido, A. Andreu, Edu H., Cos, Edu G., Maure, Dotas, Güifi, E. Gambin, Alessio Mavica, Xavi... To the people of Ireland, and this country, that are really polite, friendly and caring, and made me feel welcomed and loved and being a citizen of a good country. Finally I want to thank the guys at my local pub, as otherwise they will not allow me in next time. ;) Just kidding. When I arrived to the country I was alone, and all of you made me feel welcome, and called me by my name and understood that I was Catalan, not Spanish. In fact my first Irish friend in Ireland, Áinee, grabbed my phone and added herself to my Facebook and told me -Now we’re friends!- :) Many more, followed, like Declan. :) There I met many nice friends at the Quiz, like Pádraig. Thanks guys!. :) Go raibh míle maith agat!3 My neighbors Carmel, Mary, Margaret, Robert, that gave me chocolate bunnies for Eastern, my first umbrella, medicines, friendship and so much caring, when I was far from my family. As we are in covid madness I want to mention some of my favorite friends that keep me loved and happy: Lau, Alex, Gema and Clara, Meri and Sònia, Sandra, Cris, Edu H., Cos, Paquita, Kika, Ricky, Nessa, Deb, Lluís Urpinell, Lluís (cousin), ChocoLauri, Antonio A., Edu G., Helga Pose, George (Jawji Milla), Andrea Lc, Jordi Soler… the list is infinite. Thanks!. I really can’t mention everybody that has been nice to me, or that has helped me in one or another way. Thank you all!. 3 In Irish, means: Thanks a million Python Combat Guide 3 / 365 Carles Mateo Contact me If you want to contact me you can use the section in LeanPub. If you prefer to reach me in other way, you can contact me through my LinkedIn profile: https://www.linkedin.com/in/carlesmateo/ You can suggest new sections for the book, or problems you don’t know how to solve, report typos... You can follow me in my blog: https://blog.carlesmateo.com If you’re not satisfied If you’re not satisfied with this book, I’ve set the option in LeanPub so you can return it during the first 45 days. I think this is the only fair way to proceed. Python Combat Guide 4 / 365 Carles Mateo Index Dedication..............................................................................................................................2 Thanks...................................................................................................................................2 Contact me.............................................................................................................................4 If you’re not satisfied...............................................................................................................4 About this Combat guide......................................................................................................12 About the author...................................................................................................................12 Disclaimer.............................................................................................................................12 About coronavirus.................................................................................................................13 For the very beginners..........................................................................................................13 Hacker Rank....................................................................................................................14 About Floating point / decimals and Locale (international users).....................................15 Why Python versus Bash (reasons for SRE)........................................................................16 OOP vs Procedural code (SRE)...........................................................................................16 A practical sample of OOP for the SRE world: SSH Server Automation......................18 SSH Server improved with Paramiko SSH Libraries....................................................22 OOP small explanations recap....................................................................................23 Why not Python 2.7?............................................................................................................24 Few differences between Python 2 and 3........................................................................24 print()...............................................................................................................................25 Use a printing method instead of calling directly print()....................................................26 Use of raw_input in classes.............................................................................................26 Floats...............................................................................................................................26 Floats in Databases and Currencies................................................................................27 Things I don’t like from Python.............................................................................................30 Tools....................................................................................................................................31 IDE: PyCharm Professional.............................................................................................31 CVS: GIT.........................................................................................................................31 Unit Testing: PyTest.........................................................................................................31 Google Docs and google drive.........................................................................................32 Directories used...................................................................................................................32 File names............................................................................................................................32 File names for Classes....................................................................................................33 Classes inside files, one file per class..............................................................................33 File extensions.................................................................................................................34 File paths.........................................................................................................................36 Working with GIT..................................................................................................................37 Gitlab...............................................................................................................................37 Commit messages...........................................................................................................37 Commit messages for Gitlab.......................................................................................38 Branches..........................................................................................................................40 Branching Strategy......................................................................................................41 Git flow........................................................................................................................43 GitHub flow..................................................................................................................45 Production branch with GitLab flow.............................................................................46 Python Combat Guide 5 / 365 Carles Mateo Environment branches with GitLab flow......................................................................47 Release branches with GitLab flow.............................................................................48 Merge/Pull requests....................................................................................................49 Feature branches........................................................................................................50 Closing or Linking issues using comments in GitLab...................................................50 Removing feature branches after merge.....................................................................51 Have always backups from Master..............................................................................51 Git rebase....................................................................................................................51 Git submodules................................................................................................................52 Fetch................................................................................................................................54 Pull from the repo.............................................................................................................54 Pushing to the repo..........................................................................................................54 Keep an eye to new changes in commits through the Log section...................................55 Never ever submit to the repo local changes you want to test.........................................55 Never commit Debug Lines..............................................................................................55 Development Environment...................................................................................................56 Use Linux.........................................................................................................................56 Coding..................................................................................................................................58 General Style...................................................................................................................58 Structure......................................................................................................................58 Comments...................................................................................................................59 Use @TODO...............................................................................................................59 PEP-8..........................................................................................................................59 MT Notation.................................................................................................................60 Short Code in Methods is preferred.............................................................................62 About ' or " (apostrophe or double quote)........................................................................62 Return..............................................................................................................................63 Error code in return..........................................................................................................64 Predictability in return: Return always the same..............................................................64 Files.................................................................................................................................64 Operators.........................................................................................................................65 Tabs or spaces................................................................................................................65 Space to breath................................................................................................................66 Too much compressed code............................................................................................66 Use parentheses, Luke....................................................................................................68 Use Parenthesis on if evaluation as much as possible................................................69 Short-circuiting behavior in Chained Comparison........................................................70 Avoid using short-circuiting with methods....................................................................70 Variables and Names.......................................................................................................73 Do not use Global variables.............................................................................................73 Class variables are Static or not, depending on the context.............................................74 Declare variables on top of the Class and Method...........................................................77 Declaring variables on top in Methods and Functions.................................................77 Declaring variables on top in Classes..........................................................................78 Do not modify arrays and variables passed by reference (unless you want to do that)....81 How variables are passed to a method/function?.............................................................82 String, our immutable friend........................................................................................82 Mutable and Immutable variables....................................................................................83 Python Combat Guide 6 / 365 Carles Mateo Class Names...................................................................................................................84 Class Constants...............................................................................................................84 Comparing with constants................................................................................................85 Properties........................................................................................................................85 Properties and variables defined on top of the Method....................................................85 Functions and Methods....................................................................................................86 Do not print inside classes, use the capacity of Python to return more than a parameter instead.........................................................................................................................86 Warning: Do not repeat Method of Function names....................................................87 Function/Method Parameters with a default value (Keywords)....................................87 Keyword Argument for a Method/Function..................................................................88 Type Hinting and Annotations.....................................................................................91 Lists, Arrays, Dicts, Tuple................................................................................................93 Mixed power of Lists, Dicts..........................................................................................93 Dynamic grow..............................................................................................................94 “in” very important check to do....................................................................................94 Convenience of having a method for checking “in”......................................................95 Default dict, to save memory.......................................................................................96 Sort a List by key, example.........................................................................................97 Arrays and index -1..........................................................................................................97 Expect the unexpected (Corner cases)............................................................................98 The Happy Path..........................................................................................................98 Use getters and setters....................................................................................................99 Use update_ for methods that update Objects, not get_................................................100 Jumps of line \n to the end.............................................................................................101 If / else...........................................................................................................................102 Else in loops..............................................................................................................103 Careful String evaluations.........................................................................................103 Calling variables by name in Methods or Functions.......................................................104 If for text properties........................................................................................................104 Avoid repeating code.....................................................................................................105 Comments.....................................................................................................................106 Python doc.....................................................................................................................106 Document......................................................................................................................106 No dead code.................................................................................................................107 We avoid Magic Methods...............................................................................................107 The power of range........................................................................................................108 Avoid using yield, avoid using lambdas..........................................................................109 Avoid using list comprehensions....................................................................................110 Avoid using Regular Expressions, unless you know what you are doing.......................110 Creating Web Applications / Web Services with Flask...................................................111 Hello World Application.............................................................................................111 Another simple Web Flask Application......................................................................114 A more cool Flask Application...................................................................................115 For using in Production..............................................................................................119 Files for holding Data Structures and Databases...........................................................120 Collation....................................................................................................................120 Data types in the Database.......................................................................................120 Python Combat Guide 7 / 365 Carles Mateo Data serialization...........................................................................................................121 Some tricks to Scale Production....................................................................................124 A real case of Serialization in the world of the videogames.......................................124 Adding compression..................................................................................................124 Database Sharding....................................................................................................125 Advice: Do not allow Marketing to query Production Servers....................................127 Use of the Slaves for reading....................................................................................128 Load Balancing..........................................................................................................128 Redis cache...............................................................................................................128 Cache on local webservers.......................................................................................128 NGINX, CDNs...........................................................................................................129 Python Virtual Environments..........................................................................................129 Creating a Virtual Environment in Python 3...............................................................129 Creating a Virtual Environment in Python 2...............................................................131 Activating the Virtual Environment.............................................................................131 Deactivating the Virtual Environment.........................................................................131 Knowing what Python Packages are installed................................................................132 requirements.txt file........................................................................................................133 Python Threads..............................................................................................................134 Prefixes: MT Notation....................................................................................................135 Expect the Unexpected..................................................................................................137 Corner cases and assumptions......................................................................................139 Do no escape to shell to run a program, if you can do it from Python............................139 Escaping to shell, when really needed...........................................................................140 An example of things that could be wrong.................................................................140 Try to have Common Libraries for the Common Tasks..................................................144 OOP...................................................................................................................................145 Initialising the variables in the Constructor.....................................................................148 Use init_variables()........................................................................................................149 Bug fixing............................................................................................................................151 Focus, make sure you don’t do mistakes.......................................................................151 Better Quality than Quantity...........................................................................................151 Cross Testing.................................................................................................................151 Check the Logs..............................................................................................................152 Methodology.......................................................................................................................155 Methodology serves a purpose......................................................................................155 We are modern heroes..............................................................................................155 Deploying to Production.................................................................................................155 Create a Deploy or Launch or Change Request Document in the issue tracker........155 Deploying to Production using immutable Docker Containers and/or Microservices. 156 Requirements.txt.......................................................................................................158 In Dockerfile do not use FROM latest........................................................................160 Running in Production....................................................................................................160 Check for Log Errors regularly in Production Servers................................................160 Profiling..........................................................................................................................160 Forensics / Postmortem analysis...................................................................................160 Do not use Database Stored Procedures.......................................................................160 Webservices and API are good, microservices are not always the wisest choice..........161 Python Combat Guide 8 / 365 Carles Mateo Blue green Deployment vs Canary, and Rolling Upgrade..............................................162 Automated Code Testing Quick Overview..........................................................................165 Unit Testing Framework.................................................................................................165 py.test versus unittest?..............................................................................................165 Tests are not optional....................................................................................................165 Run the Unit Tests.........................................................................................................166 Run tests frequently..................................................................................................166 Do not reuse other Tests...........................................................................................166 Importance of running good Tests (Unit Testing Tests)..................................................166 Problems from Bash invoking parameters with quotes or without..............................167 Emails in the Tests.........................................................................................................168 Testing Infrastructure as Code...........................................................................................169 Managing the Team............................................................................................................170 Managing takes time......................................................................................................170 Synchronizing the knowledge of the Team.....................................................................170 Demos and Share the Pain............................................................................................170 Be always fair.................................................................................................................171 Allow to learn and creativity...........................................................................................172 New members join the Team.........................................................................................173 Kick Off Meeting........................................................................................................173 Concentrated advice for my Team.................................................................................173 Methodology for the Development, Presentations, Documents and Demos.......................174 Prepare Demos: name Objects and Variables...............................................................174 Prepare Demos: prepare it in advance and test that everything works...........................174 Take care of the details..................................................................................................174 We don’t avoid confronting problems.............................................................................175 We document and write Functional Analysis..................................................................175 Checking the Functional Analysis before starting programming.....................................176 We reply the emails.......................................................................................................176 We do Stand up Meetings..............................................................................................176 Exercises for Candidates / New Recruitments....................................................................176 Tools to discuss remotely (screener).............................................................................177 First, the FooBar exercise..............................................................................................177 Second, Power of 2 exercise.........................................................................................177 What we take in consideration from the candidate?.......................................................177 Being nice......................................................................................................................178 Fair Questions for Developers.......................................................................................178 Working with pytest Unit Testing........................................................................................179 Installing py.test in Ubuntu.............................................................................................179 Installing py.test for Python2.....................................................................................179 Installing py.test for Python3.....................................................................................180 Running the Tests in a Virtual Machine like VirtualBox..................................................180 Running the Tests in a Docker container.......................................................................180 Location of the code samples and tests.........................................................................181 Writing the first test........................................................................................................181 Using classes for testing................................................................................................182 Testing a real class...................................................................................................183 Extending our first test...............................................................................................185 Python Combat Guide 9 / 365 Carles Mateo Extending a bit more and getting errors.....................................................................186 Fixing our Test using TDD.............................................................................................189 Advantages and disadvantages of having everything in one single method...................190 Reusability.....................................................................................................................193 Adding improvements discussed....................................................................................193 Creating unit testing for DiskUtils...................................................................................195 Do not reuse variables in Unit Testing Code..................................................................197 Adding pytest to PyCharm Community Edition...............................................................197 Code Coverage..............................................................................................................200 Adding Dependency Injection........................................................................................203 Stubs and Mocks and Monkeypatching..........................................................................208 Putting all together.........................................................................................................213 Exercises...................................................................................................................221 Refactors.......................................................................................................................221 How to do a Refactor of legacy code.........................................................................222 Fixtures..........................................................................................................................223 Fixture to patch time.sleep........................................................................................223 Pytest config file conftest.py...........................................................................................226 Testing Exceptions.........................................................................................................228 Catching an expected Exception...............................................................................228 Code Coverage on Exceptions..................................................................................228 Test Helper for OS calls testing.................................................................................229 Testing STDOUT, print, and Fixture capsys...................................................................234 A Refactor from CSV reader spaghetti code to OOP with Unit Testing..........................235 CSV format................................................................................................................235 CSV reader code (spaghetti code)............................................................................236 Problems with this code spaghetti-code2.py..............................................................236 Addressing some errors more. spaghetti_code3.py...................................................239 Do not use dash in file names...................................................................................239 Use if __name__ == "__main__":..............................................................................239 Magic Attributes.........................................................................................................240 Extracting the functionality. spaghetti_code4.py........................................................241 Unit Testing for spaghetti_code4.py test_spaghetti_code4.py...................................243 Converting spaghetti_code4.py to OOP code............................................................246 The Unit Testing for the OOP code...........................................................................248 Converting a simple script to list drive to a well tested program with Unit Testing.....252 When to use Data Objects and when to use just Arrays or Dictionaries?..................290 Testing your code as Superuser and regular user.........................................................291 Running your Docker Container as non-root, regular user.............................................292 Test as root and non-root..........................................................................................292 Running Tests in actual Hardware?...........................................................................292 Tricks and Appendix...........................................................................................................293 A mix: Unit Testing on actual Hardware.........................................................................293 Creating PIP Packages..................................................................................................294 Convert a Python program to a Cron.............................................................................299 Using Crontab...........................................................................................................303 Using Python Scheduler Class and Docker...............................................................304 Catching CTRL + C inside your Python programs.....................................................306 Python Combat Guide 10 / 365 Carles Mateo