Rector - The Power of Automated Refactoring Matthias Noback and Tomas Votruba Thisbookisforsaleat http://leanpub.com/rector-the-power-of-automated-refactoring Thisversionwaspublishedon2021-06-24 ThisisaLeanpubbook.LeanpubempowersauthorsandpublisherswiththeLean Publishingprocess.LeanPublishingistheactofpublishinganin-progressebook usinglightweighttoolsandmanyiterationstogetreaderfeedback,pivotuntilyou havetherightbookandbuildtractiononceyoudo. ©2020-2021MatthiasNobackandTomasVotruba Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i ATrainer’sJourney . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i TheOtherSide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii Who’sFaultisIt? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii FromBlametoPain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii FromPaintoIdea. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii FromIdeatoFirstRealReconstructor . . . . . . . . . . . . . . . . . . . . . . iv We’veJustStarted,NowYouGetonBoard . . . . . . . . . . . . . . . . . . . iv Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v WhatisRector? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v WhoShouldReadThisBook? . . . . . . . . . . . . . . . . . . . . . . . . . . . vi AnOverviewoftheContents . . . . . . . . . . . . . . . . . . . . . . . . . . . vi AbouttheCodeSamples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii AbouttheAuthors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii MatthiasNoback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii TomasVotruba . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii WeWantYourFeedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix ProgrammaticallyModifyingPHPCode . . . . . . . . . . . . . . . . . . . . . . 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 PrimitiveWaysofModifyingCode . . . . . . . . . . . . . . . . . . . . . . . . 2 TokenizingPHPCode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 ParsingPHPTokens:theAbstractSyntaxTree . . . . . . . . . . . . . . . . . 8 ConvertingtheASTBacktoPHPCode . . . . . . . . . . . . . . . . . . . . . 14 ManipulatingtheAST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 CONTENTS NodeVisitors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 ManipulatingtheASTwithaNodeVisitor . . . . . . . . . . . . . . . . . . . 23 Built-inNodeVisitors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 ConnectingNodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 ResolvingFully-qualifiedNames. . . . . . . . . . . . . . . . . . . . . . . 31 FindingNodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 PHPToolsintheGame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 WorkingTogetherwithGiants . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 2007to2021Timeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 ThePrimaryFeature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 1.CodingStandardTools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 PHP_CodeSniffer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 PHPCSFixer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 EasyCodingStandard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 2.StaticAnalyzers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 PHP-Parser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 PHPStan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Psalm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.InstantUpgradeTools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Symfony-Upgrade-Fixer . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Rector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 4.InstantDowngradeTools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 PHP-Backporter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Galapagos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Transphpilea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 7to5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Rector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 WhentoUseWhichTool? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 IsYourProjectBareWithoutAnyTools? . . . . . . . . . . . . . . . . . . . . 47 RunRectorFirst,ThenPolishwithCodingStandards . . . . . . . . . . . . . 47 RecommendedTools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 CONTENTS CreatingYourFirstRectorRule . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 What’saRectorRule? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 CreatingaCustomRule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 ExtendingAbstractRector. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 FindingtheRightNodeClass. . . . . . . . . . . . . . . . . . . . . . . . . 58 RunningaSingleRule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 RefactoringtheMethodCallNode . . . . . . . . . . . . . . . . . . . . . . . . 64 WhatIstheTypeofaVariable? . . . . . . . . . . . . . . . . . . . . . . . 67 WhatifWeRunRectorTwice? . . . . . . . . . . . . . . . . . . . . . . . . . . 73 EffectivityBeatsPerfection . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Test-drivenRuleDevelopment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 MigratingfromDateTimetoDateTimeImmutable . . . . . . . . . . . . . . . 76 CreatingaTestClass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 AddingFirstTestFixture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 MakingtheFirstTestPass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 NarrowingtheScopeoftheRefactoring . . . . . . . . . . . . . . . . . . . . . 90 CapturingtheReturnValueofmodify() . . . . . . . . . . . . . . . . . . . . . 95 SkippingCallsOnClassesThatAreNotDateTime . . . . . . . . . . . . . . 104 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 MoreTestingTechniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 TestingMultipleRulesCombined . . . . . . . . . . . . . . . . . . . . . . . . . 108 RulesShouldBeIdempotent . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 RemovingtheCloneStep . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 ComparingReturnValuestoValidateCorrectness . . . . . . . . . . . . . . . 123 Shouldn’tYouStartwithaRuleSetTest? . . . . . . . . . . . . . . . . . . . . 125 Low-HangingFruitFirst . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 CreateaConfigurableGenericRule . . . . . . . . . . . . . . . . . . . . . . . 127 UsingaConfigurableInsteadofaCustomRule . . . . . . . . . . . . . . . . 128 MakingRuleBehaviorConfigurable . . . . . . . . . . . . . . . . . . . . . . . 131 ConfigurableRuleMakesHiddenAssumptionsExplicit . . . . . . . . . . . 135 UsingSupportingClassesinFixtures . . . . . . . . . . . . . . . . . . . . . . . 136 CONTENTS Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 ContinuousRectifying . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 TheNextMemberofYourTeam. . . . . . . . . . . . . . . . . . . . . . . . . . 143 TurningaJuniorIntoaSenioronDay1 . . . . . . . . . . . . . . . . . . . . . 143 Who’stoBlame? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 WhatDoesitLookLiketoWorkwithRectorinCI? . . . . . . . . . . . . . . 146 HowRectorisRectifyingItself . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 WorkFirst,EnjoyLater . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 RemovingBoringWorkOpensYourCreativity. . . . . . . . . . . . . . . . . 150 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 Rector+GitHubActions=NextMemberofYourTeam . . . . . . . . . . . . 152 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 WhatAreTheStepsforaRectorRun? . . . . . . . . . . . . . . . . . . . . . . 153 HowtoAddRectortoGitHubActions . . . . . . . . . . . . . . . . . . . . . . 154 1.GenericPHPSetup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 2.RectorSetup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 WhatFeedbackDoWeGet? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 HowtoWorkwithRector’sOutput? . . . . . . . . . . . . . . . . . . . . . . . 156 1.ManualChanges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 2.RunRectorLocally . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 3.LetRectorCodeforUsinPullRequests . . . . . . . . . . . . . . . . . 157 WhatisTheWorkWeWanttoAutomate? . . . . . . . . . . . . . . . . 158 MakeYourRectorWorkflowComeAlive . . . . . . . . . . . . . . . . . . . . 158 MakeRectorPushChanges . . . . . . . . . . . . . . . . . . . . . . . . . . 160 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 TheFutureofInstantUpgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 RemovingLegacy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 HowCanWeBeSoConfident? . . . . . . . . . . . . . . . . . . . . . . . 164 NoBackward-CompatibilityBreaksandNoBackward-Compatibility . . . 165 ABotThatSendsPull-RequestsDaily . . . . . . . . . . . . . . . . . . . . . . 167 InstantUpgradesonClick . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 PHPStandsOut. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 CONTENTS Epilogue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 Preface ByTomas Working with old code is painful. In any framework, in any company, for you, and for your employer. Nobody wants to ride a horse on the highway. But why are so manypeopledoingit?WhyaretheresomanylegacyPHPprojects? A Trainer’s Journey Iwasaskingmyselfthisquestion6yearsago,almosteveryday.Iwasdoingtrainings on Nette, Symfony, or Doctrine. My usual training looked like this: I came to a company, I gave a presentation about Symfony and its cool new features, we did hands-on coding, and I answered curious questions. I had a good feeling about this and feedback from participants was also positive. I gained a lot of experience from trainingsin30+differentPHPcompaniesanduseditineverynexttraining.Ithought: “That’s what trainers do, I finally made it; changing the life of others in just one or two days.” Gosh, I was proud of myself. Helping others is my life-long passion, so I wasveryhappy,andalsowellpaid.Thisfeelingwassoontobechanged. I had one training about new features in PHP 7 for a project that was running PHP 5.4.Afterayear,thesamecompanyaskedmeforafreshtrainingaboutnewfeatures inthePHPworld.Ithought:“TheymustbeonPHP7.1bynow.”Iaskedthem,where are they now, so I wouldn’t bore them with something they already know. Their answer: “This year we’ll definitely get to a new PHP version.” I asked: “Oh, you’re up to PHP 7.2 already? Wow!” They replied: “No, we’re almost at the PHP 7. That’s whyweneedyoutohelpus!” What?Mymindwasshocked.Suddenly,IrememberedanothertrainingIhad,where theyaskedforalmostthesametopicasayearbefore.No,thatcan’tberight,canit? Preface ii The Other Side A trainer’s point of view, and one or two days of training, are just one side of the story.Ademoiseasytocode.Everybodyishappybecausetheyworkon200linesof code and the progress is there. Then the developers have to go through a real code baseandmakethechangehappenthereaswell.Theyhavetoevolvetheprojectfrom 1.000.000 to 5.000.000 lines of code, help the business to grow, and add new features. Irealizedthatmytrainingsmayexpandknowledgebyshowingthehotnewsinthe PHP world or a particular framework. But barely 2% of this knowledge will last for longerthanaweekafterthetraining. Who’s Fault is It? I started to blame myself. My teaching methods were likely not very effective. I tried to improve my presentation and learn more about the project beforehand, but the results didn’t change much. Then I blamed the company business leaders. They were nice to hire a trainer, but after he or she is gone, they’d force programmers to work on that legacy code to add more features instead of refactoring it. Then I blamedprogrammers,whoarenotwillingtospendfourhoursafterworktoeducate themselves. They should be able to protest against legacy business and stop it, until thecodeiscleanandfuntoworkwith. LaterI’velearned,noneofthesetargetsareright. From Blame to Pain I stopped giving training so often because I felt like a failure. I was part of a system that didn’t make much change for the better. The system repeated the same action twice, with no gain. I couldn’t do it anymore. I mean, I physically could not do it. Whynot?Letmeexplain. When I was 8 years old, I lost a cap while playing outside. In situations like these, my father used to punish me with tasks involving lots of repetition. I always had to writelongsentences,a100times,like:“Iwillnevergooutsidewhileplayingandnot Preface iii returning back without the hat again, or I will have to go out and look for it and won’treturnuntilIfindit.” At first it was an interesting exercise in writing. After the 30 sentences though, my hand started to hurt. The text was supposed to be clear to read and without any mistake, and there were 10 more sentences left to go. I remember a Saturday, when Ispentalmost4hoursonthis,unabletodoanythingelse.Mymomwashelpingme sometimes,adding20sentencesherself. Thiswasrepeatedacoupletimesamonth.Ibecameterrifiedtodoanythingmyfather wouldn’t like. These punishments came out of the blue. Once my brother broke my father’s tool and told it was me. I said I don’t know anything about it, so I got the- 100-sentencespunishmentnottolieeveragain. Thankstothesepunishments,I’vehardcodedinmybrainthatanyrepetitionhurts me and I’ll do anything to avoid it. That’s my personality trait. It took some time butI’veputsomehealthyboundariesonthisandlearnedtouseittomyadvantage. P.S.Don’ttakemewrong:Ilovemyfatherandwehaveasafeandwarmrelationship now.I’mverygratefulforthepatience,attention,focus,andresiliencehetaughtme. ThankstohimI’mthemanIwanttobe. From Pain to Idea Do you know when you look at the code, you feel it’s wrong, but you don’t know whatitis?Ihadthisfeeling.Iknewtheremustbesomebetterwaytodothisthatis cheaper, faster and re-usable. It won’t be easy to find it, but there must be a way to solvethis. I started to share this pain and frustration in chats with my friends and I slowly builtupavision.Avisionofaworldwhereallprogrammersareonlythinkingabout newalgorithms,andwheretermslike“legacycode”and“technicaldebt”don’texists. Changingframeworksisamatterofasingleclick.Upgradesareaseasyascomposer update.Andit’sallopensource,soanyonecanuseit.