An Intelligent Portable Aerial Surveillance System: Modeling and Image Stitching by Ruixiang Du A Thesis Submitted to the Faculty of the WORCESTER POLYTECHNIC INSTITUTE in partial fulfillment of the requirements for the Degree of Master of Science in Robotics Engineering by MAY 2013 APPROVED: Professor Taskin Padir, Advisor Professor Michael A. Gennert, Committee Member Professor Xinming Huang, Committee Member Abstract Unmanned Aerial Vehicles (UAVs) have been widely used in modern warfare for surveil- lance, reconnaissance and even attack missions. They can provide valuable battlefield in- formation and accomplish dangerous tasks with minimal risk of loss of lives and personal injuries. However, existingUAVsystemsarefarfromperfecttomeetallpossiblesituations. One of the most notable situations is the support for individual troops. Besides the incapa- bility to always provide images in desired resolution, currently available systems are either too expensive for large-scale deployment or too heavy and complex for a single solder. In- telligent Portable Aerial Surveillance System (IPASS), sponsored by the Air Force Research Laboratory(AFRL),isaimedatdevelopingalow-cost,light-weightunmannedaerialvehicle that can provide sufficient battlefield intelligence for individual troops. The main contri- butions of this thesis are two-fold (1) the development and verification of a model-based flight simulation for the aircraft, (2) comparison of image stitching techniques to provide a comprehensive aerial surveillance information from multiple vision. To assist with the design and control of the aircraft, dynamical models are established at different complexity levels. Simulations with these models are implemented in Matlab to study the dynamical characteristics of the aircraft. Aerial images acquired from the three onboard cameras are processed after getting the flying platform built. How a particular image is formed from a camera and the general pipeline of the feature-based image stitching method are first introduced in the thesis. To better satisfy the needs of this application, a homography- based stitching method is studied. This method can greatly reduce computation time with very little compromise in the quality of the panorama, which makes real-time video display of the surroundings on the ground station possible. By implementing both of the meth- ods for image stitching using OpenCV, a quantitative comparison in the performance is accomplished. iii Acknowledgements I would like to express my deepest appreciation to all those who provided me the pos- sibility to complete this thesis. To start I would like to thank my advisor Professor Taskin Padir. Without his great patience, valuable guidance and persistent help, I would never have been able to finish this project and learn so much in this process. I consider it an honor to work with him in his Robotics and Intelligent Vehicles Research Laboratory. I would like to thank my committee members, Professor Michael A. Gennert and Pro- fessor Xinming Huang, for their insightful comments and suggestions. I would also like to thank the IPASS team, Adam Blumenau, Alec Ishak, Brett Limone, Zachary Mintz, Corey Russell and Adrian Sudol. Their works for the IPASS project made itpossibleformetotestandverifypartofmyresearch. Ourdiscussionandideaexchanging inspired me a lot to delve further into this project. Further more, I would like to thank my friends, Gang Li and Xianchao Long. They provided kindly help when I got in trouble during my works. Finally, I would like to thank my parents, Xintian Du and Wenxia Yu. Without their support, I would never have a chance to study at Worcester Polytechnic Institute in the United States. Their selfless love and encouragement always keeps me moving forward in my life and career. iv Contents List of Figures vi List of Tables viii 1 Introduction 1 1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.1 WASP III . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1.2 RQ-11 Raven . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1.3 RQ-16 T-Hawk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.2 Objectives of the IPASS Project . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2.1 Needs Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2.2 System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.3 Thesis Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.4 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.5 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2 System Design of IPASS 11 2.1 Mechanical Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.2 Electronic Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.3 Software Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.3.1 Embedded Software . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.3.2 Software for Ground Station . . . . . . . . . . . . . . . . . . . . . . 16 2.4 Final Product of the IPASS Project . . . . . . . . . . . . . . . . . . . . . . 17 2.5 Chapter Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3 Modeling and Simulation 19 3.1 Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.1.1 Basic Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.1.2 2D Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.1.3 3D Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.2 Matlab Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.2.1 Simulation Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.2.2 3D Animation for the Simulation . . . . . . . . . . . . . . . . . . . . 25 v 3.2.3 Simulation with the Basic Model . . . . . . . . . . . . . . . . . . . . 27 3.2.4 Simulation with the 2D Model . . . . . . . . . . . . . . . . . . . . . 28 3.2.5 Simulation with the 3D Model . . . . . . . . . . . . . . . . . . . . . 30 3.3 Chapter Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 4 Image Stitching 35 4.1 Image Formation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.2 Motion Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 4.3 Feature-based Image Stitching . . . . . . . . . . . . . . . . . . . . . . . . . 39 4.3.1 Feature-based Registration . . . . . . . . . . . . . . . . . . . . . . . 40 4.3.2 Compositing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 4.4 Homography Based Stitching . . . . . . . . . . . . . . . . . . . . . . . . . . 45 4.4.1 Homography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 4.4.2 Plane induced parallax . . . . . . . . . . . . . . . . . . . . . . . . . . 47 4.4.3 Infinite Homography . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 4.4.4 Camera Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 4.5 Comparison of the Two Image Stitching Methods . . . . . . . . . . . . . . . 55 5 Conclusion and Future Work 61 Bibliography 63 A Appendix A: Matlab Simulation Code for the 3D Model 65 A.1 simulation.m . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 A.2 ipass.m . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 B Appendix B: C++ Code for Homography-based Image Stitching Method 74 B.1 homostitch.h . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 B.2 homostitch.cpp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 vi List of Figures 1.1 WASP III [1] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 RQ-11 Raven [2] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.3 RQ16 T-Hawk [3] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.1 Two Prototypes of the Chassis . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.2 Final Design of the Chassis . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.3 Electronic Box and Cameras. . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.4 System Diagram of the System . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.5 Ground Station Software Developed by IPASS Team . . . . . . . . . . . . . 16 2.6 The Whole Intelligent Portable Aerial Surveillance System . . . . . . . . . 17 2.7 Image Stitching Result by IPASS team . . . . . . . . . . . . . . . . . . . . . 18 3.1 Force and Moment Analysis of the Basic Model . . . . . . . . . . . . . . . . 19 3.2 Force and Moment Analysis of the 2D Model . . . . . . . . . . . . . . . . . 21 3.3 Coordinate System of the 3D Model . . . . . . . . . . . . . . . . . . . . . . 22 3.4 Mechanical Model of the Aircraft in Sketchup . . . . . . . . . . . . . . . . . 26 3.5 Virtual Reality Scene in vrbuild2 . . . . . . . . . . . . . . . . . . . . . . . . 26 3.6 PD Control: desired value z=10, psi=0.785 (45 degrees) . . . . . . . . . . . 27 3.7 Simulation with Basic Model . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.8 Simulation with 2D Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.9 Flight Trajectory with Different values of the Offset e . . . . . . . . . . . . 29 3.10 Simulation of the Aircraft with Only Rotation about Center Axis . . . . . . 30 3.11 Simulation of the Aircraft with Only Offset of the Mass Center . . . . . . . 31 3.12 Simulation of the Aircraft when τ = 0.15 and e=0.01m . . . . . . . . . . . . 32 3.13 Simulation of the Aircraft when τ = 0.35 and e=0.01m . . . . . . . . . . . . 33 3.14 Flight Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 4.1 Pinhole Camera Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 4.2 Four Coordinates in the Image Formation . . . . . . . . . . . . . . . . . . . 37 4.3 Basic set of 2D planar transformations [4] . . . . . . . . . . . . . . . . . . . 38 4.4 Hierarchy of 2D coordinate transformations [4] . . . . . . . . . . . . . . . . 39 4.5 Pipeline of Feature-based Stitching Method from OpenCV Library . . . . . 40 4.6 Test Frame with Two Microsoft Webcams . . . . . . . . . . . . . . . . . . . 41 vii 4.7 Original Images Taken by the Two Cameras . . . . . . . . . . . . . . . . . . 42 4.8 SURF Feature Points in the Two Images . . . . . . . . . . . . . . . . . . . . 43 4.9 Matching of SURF Feature Points . . . . . . . . . . . . . . . . . . . . . . . 44 4.10 Stitching Result with a Planar Compositing Surface . . . . . . . . . . . . . 45 4.11 Stitching Result with a Spherical Compositing Surface . . . . . . . . . . . . 46 4.12 Homography between Two Image Planes [5] . . . . . . . . . . . . . . . . . . 47 4.13 Transformations of Three Cameras with Homography . . . . . . . . . . . . 48 4.14 Chessboard with Corners Marked by Color Circles and Lines . . . . . . . . 49 4.15 Image Alignment using Homography (without misalignment) . . . . . . . . 50 4.16 Image Alignment using Homography (with misalignment) . . . . . . . . . . 51 4.17 Distortion of a Camera: (a) No Distortion (b)Barrel Distortion (c) Pincush- ion Distortion [6] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.18 Distortion Correction: (Left) distorted image (Right) undistorted image . . 54 4.19 Image Alignment with Infinite Homography . . . . . . . . . . . . . . . . . . 56 4.20 Simplified Pipelines of Feature-based Method and Homography-based Method 57 4.21 GUI of the Application for Image Stitching . . . . . . . . . . . . . . . . . . 58 4.22 ComparisonofPanoramas-top: feature-basedmethod,bottom: homography- based method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 viii List of Tables 1.1 General Characteristics of WASP III [1] . . . . . . . . . . . . . . . . . . . . 2 1.2 General Characteristics of RQ-11 Raven [2] . . . . . . . . . . . . . . . . . . 4 1.3 General Characteristics of RQ-16 T-Hawk (adapted from [3], [7] and [8]) . . 4 2.1 List of Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 4.1 Update Frequency of the Two Stitching Methods . . . . . . . . . . . . . . . 58 1 Chapter 1 Introduction 1.1 Background UnmannedAerialVehicles(UAVs)havebeenwidelyusedinavarietyofareasinmodern world since their first appearance in early nineties. With the help of the achievements in technology fields such as automatic control, autonomous navigation and communication, now UAVs can accomplish various kinds of tasks that once required a human pilot to do onboard. Since the use of UAVs eliminates the possibilities of loss of lives and personal injuries, UAVs are especially suitable for tasks that need to be performed in dangerous environments. Currently, UAVs are deployed predominantly for military applications. They are mostly usedforreconnaissancetoprovidebattlefieldintelligenceandtheattacktoobjectsinspecial areas. This thesis focuses on the UAVs that are designed for reconnaissance missions. A large number of models for this purpose have been developed so far. These models are classified by different standards, such as scale and flight time. Since the research has a concentration on small scale UAVs for reconnaissance, several such existing models are briefly introduced below. 2 Figure 1.1: WASP III [1] Primary function Reconnaissance and surveillance with low-altitude operation Contractor Aerovironment, Inc. (Increment III) Power plant Electric motor, rechargeable lithium ion batteries Wingspan 28.5 inches (72.3 cm) Length 10 inches (25.4 cm) Weight (air vehicle) 1 pound (453 grams) Weight (total system) 14.4 pounds (6.53 kilograms) Speed 20 - 40 mph Operating altitude From 150 to 500+ feet above ground level (to 152+ meters) Altitude 1,000 feet System Cost approximately $49,000 (2006 dollars) Payload High resolution, day/night camera Table 1.1: General Characteristics of WASP III [1]