Feng Duan Wenyu Li Ying Tan Intelligent Robot Implementation and Applications Intelligent Robot (cid:129) (cid:129) Feng Duan Wenyu Li Ying Tan Intelligent Robot Implementation and Applications FengDuan WenyuLi CollegeofArtificialIntelligence CollegeofArtificialIntelligence NankaiUniversity NankaiUniversity Tianjin,China Tianjin,China YingTan CollegeofArtificialIntelligence NankaiUniversity Tianjin,China ISBN978-981-19-8252-1 ISBN978-981-19-8253-8 (eBook) https://doi.org/10.1007/978-981-19-8253-8 JointlypublishedwithBeijingHuazhangGraphics&InformationCo.,Ltd,ChinaMachinePress TheprinteditionisnotforsaleinChina(Mainland).CustomersfromChina(Mainland)pleaseorderthe printbookfrom:BeijingHuazhangGraphics&InformationCo.,Ltd,ChinaMachinePress. ©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNatureSingapore PteLtd.2023 Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether thewholeorpartofthematerialisconcerned,specificallytherightsofreprinting,reuseofillustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilarmethodologynowknownorhereafterdeveloped. Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant protectivelawsandregulationsandthereforefreeforgeneraluse. Thepublishers,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthis book are believed to be true and accurate at the date of publication. Neither the publishers nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publishers remain neutral with regardtojurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations. ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSingaporePteLtd. The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore Preface Robotics has developed rapidly since the mid-twentieth century when research on modern robots was conducted. Nowadays, robots are integrated into our work and daily life. With the development of computer, Internet, and artificial intelligence technologies,therearemoreandmoretypesofrobotswithenhancedfunctionsand animprovedexperienceofusingthem. Atthesametime,newadvancesinroboticsresearchanddevelopmentcontinueto be made. In particular, the Robot Operating System (ROS) is of great strategic importancetothedevelopmentoftheintelligentroboticsindustry.Robotoperating systemcanprovideaunifiedplatformforrobotdevelopment,sothatmoreuserscan conveniently study and verify robot algorithms and develop robot applications on thisplatform,whichgreatlypromotesthedevelopmentofrobottechnology.In2010, WillowGaragereleasedtheopen-sourcerobotoperatingsystemROS(RobotOper- atingSystem).Ratherthanbeinganoperatingsystem,ROSisadistributed,modular open-source software framework.Withtheadvantages ofpeer-to-peer design,pro- gramming language independence, and open source, ROS has become a new hot spotforlearninganduseinroboticsresearch. TheLaboratoryofIntelligentPerceptionandHuman–RobotInteraction,College ofArtificialIntelligence,NankaiUniversity,wheretheauthors’workisdedicatedto theresearchofintelligentrobots.Thisbookisbasedonourlong-termexperiencein developing robots using ROS, and we hope to provide a useful reference book for university students who are interested in learning intelligent robotics and technical staffwhoareengagedinintelligentrobotdevelopmentwork. Thisbookisdividedintothreeparts:thefirstpartmainlyintroducesthebasicsof robotics,includingthedefinitionofrobotics,developmenthistory,keytechnologies, frameworkanduseofROS,etc.;thesecondpartintroduceshowtodeveloparobot withrelativelycompletefunctionsintermsofrobothardwareandsoftwarecompo- sition,visionfunctionimplementation,autonomousnavigationfunctionimplemen- tation,voiceinteractionfunctionimplementation,graspingfunctionimplementation, etc.;thethirdpartgivescomprehensivecaseswithdifferentapplicationscenariosto showhowtodeveloprobotswithdifferentfunctions. v vi Preface The cases in this book are based on the robot program that the author's team participated in and won the RoboCup Robotics World Open. Beginners following the explanations in this book and practicing with the hands-on resources that accompany this book can not only master the advanced concepts of the software framework involved in robot development but also develop a fully functional intelligentrobotstepbystep. Tianjin,China FengDuan WenyuLi YingTan Contents PartI BasicKnowledge 1 OverviewofRobots. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1 DefinitionandClassificationofRobots. . . . . . . . . . . . . . . . . . . 3 1.1.1 DefinitionofRobots. . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1.2 ClassificationofServiceRobots. . . . . . . . . . . . . . . . . 4 1.2 TheEvolutionofModernRobots. . . . . . . . . . . . . . . . . . . . . . . 5 1.2.1 EarlyStagesofModernRobotsResearch. . . . . . . . . . . 5 1.2.2 The1970s. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2.3 The1980s. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2.4 The1990s. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.2.5 Twenty-FirstCentury. . . . . . . . . . . . . . . . . . . . . . . . . 15 1.3 CompositionoftheRobots. . . . . . . . . . . . . . . . . . . . . . . . . . . 23 1.3.1 ActuatorsfortheRobot. . . . . . . . . . . . . . . . . . . . . . . 24 1.3.2 DriveUnits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.3.3 SensingDevices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.3.4 ControlSystems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.3.5 IntelligentSystems. . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.3.6 IntelligentHuman-MachineInterfaceSystems. . . . . . . 26 1.4 KeyTechnologiesforRobots. . . . . . . . . . . . . . . . .. . . . . . . . . 26 1.4.1 AutonomousMobileTechnologies. . . . . . . . . . . . . . . 26 1.4.2 PerceptualTechnologies. . . . . . . . . . . . . . . . . . . . . . . 27 1.4.3 IntelligentDecision-MakingandControl Technologies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 1.4.4 CommunicationTechnologies. . . . . . . . . . . . . . . . . . . 27 1.5 TrendsinRobots. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 1.5.1 HierarchicalandHumanizedHuman-Computer Interaction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 vii viii Contents 1.5.2 InteractionIntelligencewiththeEnvironment. . . . . . . . 28 1.5.3 NetworkingofResourceUtilization. . . . . . . . . . . . . . . 28 1.5.4 Standardization,ModularizationandSystematization ofDesignandProduction. . . . . . . . . . . . . . . . . . . . . . 29 FurtherReading. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2 GettingStartedwithROS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 2.1 IntroductiontoROS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.1.1 WhyUseROS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.1.2 WhatIsROS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.1.3 DifferencesBetweenROSandComputerOperating Systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 2.1.4 KeyFeaturesofROS. . . . . . . . . . . . . . . . . . . . . . . . . 33 2.2 InstallationandUninstallationofROS. . . . . . . . . . . . . . . . . . . 35 2.2.1 VersionsofROS. . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 2.2.2 InstallingandConfiguringROSIndigo. . . . . . . . . . . . 35 2.2.3 InstallingandConfiguringROSMelodic. . . . . . . . . . . 38 2.2.4 UninstallationofROS. . . . . . . . . . . . . . . . . . . . . . . . 40 2.3 ResourcesforFurtherLearning. . . . . . . . . . . . . . . . . . . . . . . . 40 FurtherReading. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3 TheFrameworkandFundamentalUseofROS. . . . . . . . . . . . . . . . 43 3.1 ROSFramework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.1.1 FileSystemLevel. . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.1.2 CalculationofGraphLevels. . . . . . . . . . . . . . . . . . . . 44 3.1.3 CommunityLevel. . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.2 BasicsofROSUse. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.2.1 OverviewofCatkin. . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.2.2 WorkspacesandTheirCreationMethods. . . . . . . . . . . 47 3.2.3 CreatingROSProjectPackages. . . . . . . . . . . . . . . . . . 50 3.2.4 CompilingtheROSProjectPackage. . . . . . . . . . . . . . 51 3.2.5 CreatingROSNodes. . . . . . . . . . . . . . . . . . . . . . . . . 51 3.2.6 CompilingandRunningROSNodes. . . . . . . . . . . . . . 52 3.2.7 UseofRoslaunch. . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.2.8 CreatingROSMessagesandServices. . . . . . . . . . . . . 55 3.2.9 WritingaSimpleMessagePublisherandSubscriber (C++Implementation). . . . . . . . . . . . . . . . . . . . . . . . 59 3.2.10 WritingaSimpleMessagePublisherandSubscriber (PythonImplementation). . . . . . . . . . . . . . . . . . . . . . 61 3.2.11 TestingMessagePublisherandSubscriber. . . . . . . . . . 63 3.2.12 WritingaSimpleServerandClient(C++ implementation). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 3.2.13 WritingaSimpleServerandClient(Python Implementation). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 3.2.14 TestingaSimpleServerandClient. . . . . . . . . . . . . . . 68 FurtherReading. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 Contents ix 4 ROSDebugging. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 4.1 CommonCommandsforROSDebugging. . . . . . . . . . . . . . . . 71 4.2 CommonToolsforROSDebugging. . . . . . . . . . . . . . . . . . . . 73 4.2.1 Usingrqt_consoletoModifytheDebugLevel atRuntime. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 4.2.2 UsingroswtftoDetectPotentialProblemsinthe Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 4.2.3 DisplayingNodeStateGraphsUsingrqt_graph. . . . . . 76 4.2.4 PlottingScalarDataUsingrqt_plot. . . . . . . . . . . . . . . 78 4.2.5 Displaying2DImagesUsingimage_view. . . . . . . . . . 78 4.2.6 3DDataVisualizationUsingrqt_rviz(rviz). . . . . . . . . 80 4.2.7 RecordingandPlayingBackDataUsingrosbagand rqt_bag. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 4.2.8 rqtPluginsandrxApplications. . . . . . . . . . . . . . . . . . 84 4.3 SummaryofBasicROSCommands. . . . . . . . . . . . . . . . . . . . . 84 4.3.1 CreatingaROSWorkspace. . . .. . . . .. . . . . .. . . . .. 85 4.3.2 PackageRelatedOperations. . . . . . . . . . . . . . . . . . . . 85 4.3.3 RelatedOperationsofNodes. . . . . . . . . . . . . . . . . . . . 86 4.3.4 RelatedOperationsoftheTopic. . . . . . . . . . . . . . . . . 87 4.3.5 RelatedOperationsoftheService. . . . . . . . . . . . . . . . 88 4.3.6 RelatedOperationsofrosparam. . . . . . . . . . . . . . . . . . 89 4.3.7 Bag-RelatedOperations. . . . . . . . . . . . . . . . . . . . . . . 89 4.3.8 RelatedOperationsofrosmsg. . . . . . . . . . . . . . . . . . . 90 4.3.9 RelatedOperationsofrossrv. . . . . . . . . . . . . . . . . . . . 91 4.3.10 OtherCommandsofROS. . . . . . . . . . . . . . . . . . . . . . 91 PartII ImplementationofCoreRobotFunctions 5 InstallationandInitialUseoftheRobots. . . . . . . . . . . . . . . . . . . . 95 5.1 IntroductiontotheTurtlebotRobot. . . . . . . . . . . . . . . . . . . . . 95 5.2 CompositionandConfigurationoftheTurtlebotRobot Hardware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 5.3 InstallationandTestingoftheTurtlebotRobotSoftware. . . . .. 98 5.3.1 InstallingfromSource. . . . . . . . . . . . . . . . . . . . . . . . 98 5.3.2 DebInstallationMethod. . . . . . . . . . . . . . . . . . . . . . . 99 5.3.3 ConfigurationAccordingtoKobukiBase. . . . . . . . . . . 100 5.4 LaunchingTurtlebot. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 5.5 ManualControlofTurtlebotviaKeyboard. . . . . . . . . . . . . . . . 102 5.6 ControllingTurtlebotThroughScripting. . . . . . . . . . . . . . . . . . 103 5.7 MonitoringtheBatteryStatusoftheKobuki. . . . . . . . . . . . . . . 104 5.8 ExtensionstotheTurtlebotRobot. . . . . . . . . . . . . . . . . . . . . . 106 FurtherReading. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 x Contents 6 ImplementationofRobotVisionFunctions. . . . . . . . . . . . . . . . . . . 109 6.1 VisionSensors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 6.1.1 KinectVisionSensor. . . . . . . . . . . . . . . . . . . . . . . . . 110 6.1.2 PrimesenseVisionSensors. . . . . . . . . . . . . . . . . . . . . 111 6.2 DriverInstallationandTesting. . . . . . . . . . . . . . . . . . . . . . . . . 112 6.3 RunningTwoKinectsattheSameTime. . . . . . . . . . . . . . . . . . 113 6.4 RunningKinectandPrimesenseattheSameTime. . . . . . . . . . 116 6.5 RGBImageProcessingWithOpenCVinROS. . . . . . . . . . . . . 117 6.5.1 InstallingOpenCVinROS. . . . . . . . . . . . . . . . . . . . . 117 6.5.2 UsingOpenCVinROSCode. . . . . . . . . . . . . . . . . . . 118 6.5.3 UnderstandingtheROS-OpenCVConversion Architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 6.5.4 ROSNodeExample. . . . . . . . . . . . . . . . . . . . . . . . . . 121 6.6 PointCloudLibraryandItsUse. . . . . . . . . . . . . . . . . . . . . . . . 125 6.6.1 IntroductiontoPointCloudsandPointCloud Libraries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 6.6.2 DataTypesofPCL. .. . . . . .. . . . . .. . . . . .. . . . . .. 125 6.6.3 PublishandSubscribtoPointCloudMessages. . . . . . . 127 6.6.4 TutorialonHowtoUsePCLinROS. . . . . . . . . .. . . . 129 6.6.5 ASimpleApplicationofPCL-DetectingtheOpening andClosingStateofaDoor. . . . . . . . . . . . . . . . . . . . 136 FurtherReading. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 7 AdvancedImplementationofRobotVisionFunctions. . . . . . . . . . . 139 7.1 ImplementationoftheRobotFollowFunction. . . . . . . . . . . . . 139 7.1.1 TheoreticalFoundations. . . . . . . . . . . . . . . . . . . . . . . 139 7.1.2 OperationalTestingofFollowFunctions. . . . . . . . . . . 140 7.2 ImplementationoftheRobotWavingRecognitionFunction. . . 144 7.2.1 ImplementationFrameworkandDifficultiesAnalysis ofRobotHandWavingRecognition. . . . . . . . . . . . . . 145 7.2.2 FaceDetectionBasedonAdaBoostandCascade Algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 7.2.3 IdentifyingHumanHandsWiththeTemplate MatchingAlgorithm. . . . . . . . . . . . . . . . . . . . . . . . . . 149 7.2.4 SkinToneSegmentationBasedonYCrCbColor Space. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 7.2.5 OperationalTestingoftheWaveRecognition Function. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 7.3 ImplementationofObjectRecognitionandLocalization FunctionsoftheRobot. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 7.3.1 SlidingWindowTemplateMatchingMethod BasedonHueHistogram. . . . . . . . . . . . . . . . . . . . . . 152 7.3.2 ObjectLocalizationMethodsBasedonSpatial PointCloudData. . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 7.3.3 ImplementationandTestingofObjectRecognition andLocalization......................... .... 156