The Algorithm Design Manual Second Edition Steven S. Skiena The Algorithm Design Manual Second Edition 123 StevenS.Skiena DepartmentofComputerScience StateUniversityofNewYork atStonyBrook NewYork,USA [email protected] ISBN:978-1-84800-069-8 e-ISBN:978-1-84800-070-4 DOI:10.1007/978-1-84800-070-4 BritishLibraryCataloguinginPublicationData AcataloguerecordforthisbookisavailablefromtheBritishLibrary LibraryofCongressControlNumber:2008931136 (cid:2)c Springer-VerlagLondonLimited2008 Apartfromanyfairdealingforthepurposesofresearchorprivatestudy,orcriticismorreview,aspermitted undertheCopyright,DesignsandPatentsAct1988,thispublicationmayonlybereproduced,storedortrans- mitted,inanyformorbyanymeans,withthepriorpermissioninwritingofthepublishers,orinthecaseof reprographicreproductioninaccordancewiththetermsoflicensesissuedbytheCopyrightLicensingAgency. Enquiriesconcerningreproductionoutsidethosetermsshouldbesenttothepublishers. The use of registered names, trademarks, etc., in this publication does not imply, even in the absence of a specificstatement,thatsuchnamesareexemptfromtherelevantlawsandregulationsandthereforefreefor generaluse. The publisher makes no representation, express or implied, with regard to the accuracy of the information containedinthisbookandcannotacceptanylegalresponsibilityorliabilityforanyerrorsoromissionsthat maybemade. Printedonacid-freepaper SpringerScience+BusinessMedia springer.com Preface Most professional programmers that I’ve encountered are not well prepared to tacklealgorithmdesignproblems.Thisisapity,becausethetechniquesofalgorithm design form one of the core practical technologies of computer science. Designing correct, efficient, and implementable algorithms for real-world problems requires access to two distinct bodies of knowledge: • Techniques – Good algorithm designers understand several fundamental al- gorithmdesigntechniques,includingdatastructures,dynamicprogramming, depth-first search, backtracking, and heuristics. Perhaps the single most im- portantdesigntechniqueismodeling,theartofabstractingamessyreal-world application into a clean problem suitable for algorithmic attack. • Resources – Good algorithm designers stand on the shoulders of giants. Ratherthanlaboringfromscratchtoproduceanewalgorithmforeverytask, they can figure out what is known about a particular problem. Rather than re-implementing popular algorithms from scratch, they seek existing imple- mentations to serve as a starting point. They are familiar with many classic algorithmicproblems,whichprovidesufficientsourcematerialtomodelmost any application. This book is intended as a manual on algorithm design, providing access to combinatorial algorithm technology for both students and computer professionals. It is divided into two parts: Techniques and Resources. The former is a general guide to techniques for the design and analysis of computer algorithms. The Re- sources section is intended for browsing and reference, and comprises the catalog of algorithmic resources, implementations, and an extensive bibliography. vi PREFACE To the Reader I have been gratified by the warm reception the first edition of The Algorithm De- signManualhasreceivedsinceitsinitialpublicationin1997.Ithasbeenrecognized asauniqueguidetousingalgorithmictechniquestosolveproblemsthatoftenarise in practice. But much has changed in the world since the The Algorithm Design Manual was first published over ten years ago. Indeed, if we date the origins of modernalgorithm designandanalysis toabout 1970, thenroughly 30% ofmodern algorithmic history has happened since the first coming of The Algorithm Design Manual. ThreeaspectsofThe Algorithm Design Manualhavebeenparticularlybeloved: (1) the catalog of algorithmic problems, (2) the war stories, and (3) the electronic component of the book. These features have been preserved and strengthened in this edition: • TheCatalogofAlgorithmicProblems–Sincefindingoutwhatisknownabout an algorithmic problem can be a difficult task, I provide a catalog of the 75 most important problems arising in practice. By browsing through this catalog,thestudentorpractitionercanquicklyidentifywhattheirproblemis called,whatisknownaboutit,andhowtheyshouldproceedtosolveit.Toaid inproblemidentification,weincludeapairof“before”and“after”picturesfor each problem, illustrating the required input and output specifications. One perceptive reviewer called my book “The Hitchhiker’s Guide to Algorithms” on the strength of this catalog. The catalog is the most important part of this book. To update the catalog for this edition, I have solicited feedback from the world’s leading experts on each associated problem. Particular attention has been paid to updating the discussion of available software implementations for each problem. • WarStories–Inpractice,algorithmproblemsdonotariseatthebeginningof a large project. Rather, they typically arise as subproblems when it becomes clearthattheprogrammerdoesnotknowhowtoproceedorthatthecurrent solution is inadequate. To provide a better perspective on how algorithm problems arise in the real world, we include a collection of “war stories,” or tales from our experience with real problems. The moral of these stories is that algorithm design and analysis is not just theory, but an important tool to be pulled out and used as needed. Thiseditionretainsalltheoriginalwarstories(withupdatesasappropriate) plus additional new war stories covering external sorting, graph algorithms, simulated annealing, and other topics. • Electronic Component – Since the practical person is usually looking for a program more than an algorithm, we provide pointers to solid implementa- tions whenever they are available. We have collected these implementations PREFACE vii at one central website site (http://www.cs.sunysb.edu/∼algorith) for easy re- trieval. We have been the number one “Algorithm” site on Google pretty much since the initial publication of the book. Further,weproviderecommendationstomakeiteasiertoidentifythecorrect code for the job. With these implementations available, the critical issue in algorithm design becomes properly modeling your application, more so than becoming intimate with the details of the actual algorithm. This focus permeates the entire book. Equally important is what we do not do in this book. We do not stress the mathematical analysis of algorithms, leaving most of the analysis as informal ar- guments. You will not find a single theorem anywhere in this book. When more details are needed, the reader should study the cited programs or references. The goal of this manual is to get you going in the right direction as quickly as possible. To the Instructor ThisbookcoversenoughmaterialforastandardIntroductiontoAlgorithmscourse. We assume the reader has completed the equivalent of a second programming course, typically titled Data Structures or Computer Science II. A full set of lecture slides for teaching this course is available online at http://www.algorist.com.Further,Imakeavailableonlineaudioandvideolectures usingtheseslidestoteachafull-semesteralgorithmcourse.Letmehelpteachyour course, by the magic of the Internet! Thisbookstressesdesignoveranalysis.Itissuitableforbothtraditionallecture coursesandthenew“activelearning”method,wheretheprofessordoesnotlecture butinsteadguidesstudentgroupstosolverealproblems.The“warstories”provide an appropriate introduction to the active learning method. Ihavemadeseveralpedagogicalimprovementsthroughoutthebook.Textbook- oriented features include: • MoreLeisurelyDiscussion–Thetutorialmaterialinthefirstpartofthebook has been doubled over the previous edition. The pages have been devoted to more thorough and careful exposition of fundamental material, instead of adding more specialized topics. • False Starts – Algorithms textbooks generally present important algorithms as a fait accompli, obscuring the ideas involved in designing them and the subtle reasons why other approaches fail. The war stories illustrate such de- velopment on certain applied problems, but I have expanded such coverage into classical algorithm design material as well. • Stop and Think – Here I illustrate my thought process as I solve a topic- specific homework problem—false starts and all. I have interspersed such viii PREFACE problem blocks throughout the text to increase the problem-solving activity of my readers. Answers appear immediately following each problem. • More and Improved Homework Problems – This edition of The Algorithm Design Manual has twice as many homework exercises as the previous one. Exercises that proved confusing or ambiguous have been improved or re- placed. Degree of difficulty ratings (from 1 to 10) have been assigned to all problems. • Self-Motivating Exam Design – In my algorithms courses, I promise the stu- dents that all midterm and final exam questions will be taken directly from homeworkproblemsinthisbook.Thisprovidesa“student-motivatedexam,” sostudentsknowexactlyhowtostudytodowellontheexam.Ihavecarefully pickedthequantity,variety,anddifficultyofhomeworkexercisestomakethis work; ensuring there are neither too few or too many candidate problems. • Take-Home Lessons – Highlighted “take-home” lesson boxes scattered throughout the text emphasize the big-picture concepts to be gained from the chapter. • Links to Programming Challenge Problems – Each chapter’s exercises will contain links to 3-5 relevant “Programming Challenge” problems from http://www.programming-challenges.com. These can be used to add a pro- gramming component to paper-and-pencil algorithms courses. • More Code, Less Pseudo-code–Morealgorithmsinthisbookappearascode (written in C) instead of pseudo-code. I believe the concreteness and relia- bility of actual tested implementations provides a big win over less formal presentations for simple algorithms. Full implementations are available for study at http://www.algorist.com . • Chapter Notes – Each tutorial chapter concludes with a brief notes section, pointing readers to primary sources and additional references. Acknowledgments Updatingabookdedicationaftertenyearsfocusesattentionontheeffectsoftime. Since the first edition, Renee has become my wife and then the mother of our two children, Bonnie and Abby. My father has left this world, but Mom and my brothers Len and Rob remain a vital presence in my life. I dedicate this book to my family, new and old, here and departed. I would like to thank several people for their concrete contributions to this new edition. Andrew Gaun and Betson Thomas helped in many ways, particularly in building the infrastructure for the new http://www.cs.sunysb.edu/∼algorithand dealing with a variety of manuscript preparation issues. David Gries offered valu- ablefeedbackwellbeyondthecallofduty.HimanshuGuptaandBinTangbravely PREFACE ix taught courses using a manuscript version of this edition. Thanks also to my Springer-Verlag editors, Wayne Wheeler and Allan Wylde. A select group of algorithmic sages reviewed sections of the Hitchhiker’s guide, sharing their wisdom and alerting me to new developments. Thanks to: Ami Amir, Herve Bronnimann, Bernard Chazelle, Chris Chu, Scott Cotton,YefimDinitz,KomeiFukuda,MichaelGoodrich,LennyHeath, Cihat Imamoglu, Tao Jiang, David Karger, Giuseppe Liotta, Albert Mao, Silvano Martello, Catherine McGeoch, Kurt Mehlhorn, Scott A. Mitchell, Naceur Meskini, Gene Myers, Gonzalo Navarro, Stephen North,JoeO’Rourke,MikePaterson,TheoPavlidis,SethPettie,Michel Pocchiola, Bart Preneel, Tomasz Radzik, Edward Reingold, Frank Ruskey, Peter Sanders, Joao Setubal, Jonathan Shewchuk, Robert Skeel, Jens Stoye, Torsten Suel, Bruce Watson, and Uri Zwick. Several exercises were originated by colleagues or inspired by other texts. Re- constructingtheoriginalsourcesyearslatercanbechallenging,butcreditsforeach problem (to the best of my recollection) appear on the website. It would be rude not to thank important contributors to the original edition. Ricky Bradley and Dario Vlah built up the substantial infrastructure required for the WWW site in a logical and extensible manner. Zhong Li drew most of the catalog figures using xfig. Richard Crandall, Ron Danielson, Takis Metaxas, Dave Miller, Giri Narasimhan, and Joe Zachary all reviewed preliminary versions of the first edition; their thoughtful feedback helped to shape what you see here. Much of what I know about algorithms I learned along with my graduate students. Several of them (Yaw-Ling Lin, Sundaram Gopalakrishnan, Ting Chen, Francine Evans, Harald Rau, Ricky Bradley, and Dimitris Margaritis) are the real heroes of the war stories related within. My Stony Brook friends and algorithm colleagues Estie Arkin, Michael Bender, Jie Gao, and Joe Mitchell have always been a pleasure to work and be with. Finally, thanks to Michael Brochstein and the rest of the city contingent for revealing a proper life well beyond Stony Brook. Caveat It is traditional for the author to magnanimously accept the blame for whatever deficiencies remain. I don’t. Any errors, deficiencies, or problems in this book are somebody else’s fault, but I would appreciate knowing about them so as to deter- mine who is to blame. Steven S. Skiena Department of Computer Science Stony Brook University Stony Brook, NY 11794-4400 http://www.cs.sunysb.edu/∼skiena April 2008 Contents I Practical Algorithm Design 1 1 Introduction to Algorithm Design 3 1.1 Robot Tour Optimization . . . . . . . . . . . . . . . . . . . . . . 5 1.2 Selecting the Right Jobs . . . . . . . . . . . . . . . . . . . . . . . 9 1.3 Reasoning about Correctness . . . . . . . . . . . . . . . . . . . . 11 1.4 Modeling the Problem . . . . . . . . . . . . . . . . . . . . . . . . 19 1.5 About the War Stories . . . . . . . . . . . . . . . . . . . . . . . . 22 1.6 War Story: Psychic Modeling . . . . . . . . . . . . . . . . . . . . 23 1.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2 Algorithm Analysis 31 2.1 The RAM Model of Computation . . . . . . . . . . . . . . . . . . 31 2.2 The Big Oh Notation . . . . . . . . . . . . . . . . . . . . . . . . . 34 2.3 Growth Rates and Dominance Relations . . . . . . . . . . . . . . 37 2.4 Working with the Big Oh . . . . . . . . . . . . . . . . . . . . . . 40 2.5 Reasoning About Efficiency . . . . . . . . . . . . . . . . . . . . . 41 2.6 Logarithms and Their Applications . . . . . . . . . . . . . . . . . 46 2.7 Properties of Logarithms . . . . . . . . . . . . . . . . . . . . . . . 50 2.8 War Story: Mystery of the Pyramids . . . . . . . . . . . . . . . . 51 2.9 Advanced Analysis (*) . . . . . . . . . . . . . . . . . . . . . . . . 54 2.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 3 Data Structures 65 3.1 Contiguous vs. Linked Data Structures. . . . . . . . . . . . . . . 66