Persistent Aerial Tracking Thesis by Matthias Mueller In Partial Fulfillment of the Requirements For the Degree of Masters of Science in Electrical Engineering King Abdullah University of Science and Technology, Thuwal, Kingdom of Saudi Arabia 'April 13th, 2016 Matthias Mueller All Rights Reserved 2 The thesis of Matthias Mueller is approved by the examination committee Committee Chairperson: Bernard Ghanem Committee Member: Jeff Shamma Committee Member: Peter Wonka 3 ABSTRACT Peristent Aerial Tracking Matthias Mueller In this thesis, we propose a new aerial video dataset and benchmark for low altitude UAV target tracking, as well as, a photo-realistic UAV simulator that can be coupled with tracking methods. Our benchmark provides the first evaluation of many state- of-the-art and popular trackers on 123 new and fully annotated HD video sequences captured from a low-altitude aerial perspective. Among the compared trackers, we determine which ones are the most suitable for UAV tracking both in terms of track- ing accuracy and run-time. We also present a simulator that can be used to evaluate tracking algorithms in real-time scenarios before they are deployed on a UAV ”in the field”, as well as, generate synthetic but photo-realistic tracking datasets with free ground truth annotations to easily extend existing real-world datasets. Both the benchmark and simulator will be made publicly available to the vision community to further research in the area of object tracking from UAVs. Additionally, we propose a persistent, robust and autonomous object tracking system for unmanned aerial vehi- cles (UAVs) called Persistent Aerial Tracking (PAT). A computer vision and control strategy is applied to a diverse set of moving objects (e.g. humans, animals, cars, boats, etc.) integrating multiple UAVs with a stabilized RGB camera. A novel strat- egy is employed to successfully track objects over a long period, by ’handing over the camera’ from one UAV to another. We integrate the complete system into an off- 4 the-shelf UAV, and obtain promising results showing the robustness of our solution in real-world aerial scenarios. Keywords: Aerial Surveillance, Object Tracking, Autonomous UAV Control, Dataset, Benchmark, Simulator, Unreal Engine 4 5 ACKNOWLEDGEMENTS First and foremost, I would like to express my gratitude to my Lord and Saviour Jesus Christ for giving my life a purpose and blessing me with a great family, friends and talents enabling me to do this work. I would also like to express my sincere appreciation and gratitude to KAUST for the financial support, and to my supervisor Prof. Bernard Ghanem for his academic support and guidance. He has been a great source of knowledge and has inspired me with valuable ideas. Furthermore, I extend my deepest appreciation to Neil Smith for his patience, encour- agement, continued support and outstanding work on the simulator. He was always ready to help with technical details and willing to stay up all night with me during crunch times. In addition, I would like to thank Luca Passone, Ryan Boekeloo and Mohammed Karkadan from FalconViz for risking their lives by helping me with the real-world tests of the tracking system. I would also like to acknowledge the work of Gopal Sharma on the reinitialization module. Moreover, I would like to thank Adel Bibi and the rest of the IVUL group as well as my other colleagues and friends at King Abdullah University of Science and Technol- ogy, for their encouragement, support and all the fun times. I am also very grateful to Prof. Jeff Shamma and Prof. Peter Wonka for being part of the Master Thesis committee. Finally, I would also like to thank my wife for her continued patience, support and encouragement. I want to extend my deepest love and gratitude to her, my two beautiful daughters and my parents by dedicating this work to them. 6 TABLE OF CONTENTS Examination Committee Approval 2 Abstract 3 Acknowledgements 5 List of Abbreviations 9 List of Figures 10 List of Tables 12 1 Introduction 13 1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.2.1 UAV Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.2.2 UAV Simulator . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.2.3 Generic Object Tracking . . . . . . . . . . . . . . . . . . . . . 18 1.2.4 UAV Tailored Tracking . . . . . . . . . . . . . . . . . . . . . . 19 1.3 Outline of the thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2 Benchmark - Offline Evaluation 22 2.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.1.1 Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.1.2 Acquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.1.3 Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.1.4 Long-term tracking . . . . . . . . . . . . . . . . . . . . . . . . 26 2.2 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.3 Evaluation Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.4 Experiments - Benchmark Evaluation . . . . . . . . . . . . . . . . . . 30 2.4.1 Overall Performance . . . . . . . . . . . . . . . . . . . . . . . 30 7 2.4.2 Per-Attribute Performance . . . . . . . . . . . . . . . . . . . . 31 2.4.3 Per-Video Performance . . . . . . . . . . . . . . . . . . . . . . 35 2.4.4 Sensitivity to Frame Rate . . . . . . . . . . . . . . . . . . . . 36 2.4.5 Long-Term Tracking . . . . . . . . . . . . . . . . . . . . . . . 37 2.4.6 Spatial Robustness . . . . . . . . . . . . . . . . . . . . . . . . 38 2.4.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3 Simulator - Online Evaluation 40 3.1 Simulator Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.1.1 UAV Physics Simulation and Control . . . . . . . . . . . . . . 41 3.1.2 Frame Capture, Object Segmentation, Flight Logging . . . . . 42 3.1.3 MATLAB/C++ Integration . . . . . . . . . . . . . . . . . . . 42 3.1.4 Visual Servoing . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.2 Evaluation Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.2.1 Qualitative Tracker Performance Visualization . . . . . . . . . 44 3.3 Experiments - Simulator Evaluation . . . . . . . . . . . . . . . . . . . 45 3.3.1 Sensitivity to Frame Rate . . . . . . . . . . . . . . . . . . . . 47 3.3.2 Sensitivity to UAV Response . . . . . . . . . . . . . . . . . . . 47 3.3.3 Long-Term Tracking . . . . . . . . . . . . . . . . . . . . . . . 48 3.3.4 Per-Attribute Performance . . . . . . . . . . . . . . . . . . . . 48 3.3.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 4 Integration - Aerial Tracking System 51 4.1 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 4.2 Onboard Visual Tracking System . . . . . . . . . . . . . . . . . . . . 53 4.3 Tracking Control System . . . . . . . . . . . . . . . . . . . . . . . . . 54 4.4 Handover System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 4.5 Tracker Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 4.5.1 Evaluation Setup . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.5.2 Evaluation Results . . . . . . . . . . . . . . . . . . . . . . . . 60 4.5.3 Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 4.6 Target Initialization Module . . . . . . . . . . . . . . . . . . . . . . . 65 4.7 Bounding Box Evaluation and Re-initialization Module . . . . . . . . 67 4.8 Experiments - System Evaluation . . . . . . . . . . . . . . . . . . . . 70 4.8.1 Regular Motion . . . . . . . . . . . . . . . . . . . . . . . . . . 70 4.8.2 Rapid/Erratic Motion . . . . . . . . . . . . . . . . . . . . . . 71 4.8.3 Robustness to Occlusion . . . . . . . . . . . . . . . . . . . . . 71 8 4.8.4 Camera Handover . . . . . . . . . . . . . . . . . . . . . . . . . 72 4.8.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 5 Conclusion 73 5.1 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 5.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 References 76 Appendix 86 9 LIST OF ABBREVIATIONS FCS Flight Control System GCS Ground Control Station OPE One Pass Evaluation OTB Online Tracking Benchmark PAT Persistent Aerial Tracking SRE Spatial Robustness Evaluation TRE Temporal Robustness Evaluation UAV Unmanned Aerial Vehicle UE4 Unreal Engine 4 10 LIST OF FIGURES 2.1 Histogram of aspect ratio, bounding box size and sequence length across three datasets: OTB100, TC128, and UAV123 (ours). . . . . . 24 2.2 First frame of selected sequences from UAV123 dataset. . . . . . . . . 25 2.3 Attribute distribution across the UAV123 dataset and a comparison of key attributes with OTB100. . . . . . . . . . . . . . . . . . . . . . . . 27 2.4 Dataset Difficulty Comparison. . . . . . . . . . . . . . . . . . . . . . 28 2.5 Precision and success plots for OPE on UAV123. . . . . . . . . . . . . 31 2.6 Precision and success plots for OPE on OTB100. . . . . . . . . . . . 31 2.7 Precision and success plots for OPE on TC128. . . . . . . . . . . . . 31 2.8 OPE precision plots by attribute. . . . . . . . . . . . . . . . . . . . . 32 2.9 OPE success plots by attribute. . . . . . . . . . . . . . . . . . . . . . 33 2.10 OPE Precision (top) and Success (bottom) per video for UAV123. . . 35 2.11 OPE Precision (top) and Success (bottom) per video for OTB100. . . 36 2.12 Precision and success plots for OPE on UAV123@10fps. . . . . . . . . 37 2.13 Precision and success plots for OPE on UAV20L. . . . . . . . . . . . 38 2.14 Precision and success plots for SRE on UAV123. . . . . . . . . . . . . 38 3.1 Third person view of simulator environment. . . . . . . . . . . . . . . 41 3.2 Synthetic dataset generation and online tracker evaluation using the proposed simulator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.3 Third person view of object being tracked by multiple UAVs corre- sponding to different trackers. . . . . . . . . . . . . . . . . . . . . . . 45 3.4 Trajectory of tracker controlled UAV over the period of the simulation andmultipletrackersboundingboxeslayeredoverthetrackedsynthetic frame. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 4.1 The two classes of unmanned aerial vehicles integrated with the on- board vision based flight control system. . . . . . . . . . . . . . . . . 52 4.2 Hardware and software of FCS and GCS. . . . . . . . . . . . . . . . . 53 4.3 Flowchart of the onboard visual tracking system. . . . . . . . . . . . 55