AUTOMATIC ALIGNMENT OF 3D MULTI-SENSOR POINT CLOUDS RAVI ANCIL PERSAD A DISSERTATION SUBMITTED TO THE FACULTY OF GRADUATE STUDIES IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY GRADUATE PROGRAMME IN EARTH AND SPACE SCIENCE YORK UNIVERSITY TORONTO, ONTARIO AUGUST 2017 © Ravi Ancil Persad, 2017 ii Abstract Automatic 3D point cloud alignment is a major research topic in photogrammetry, computer vision and computer graphics. In this research, two keypoint feature matching approaches have been developed and proposed for the automatic alignment of 3D point clouds, which have been acquired from different sensor platforms and are in different 3D conformal coordinate systems. The first proposed approach is based on 3D keypoint feature matching. First, surface curvature information is utilized for scale-invariant 3D keypoint extraction. Adaptive non-maxima suppression (ANMS) is then applied to retain the most distinct and well- distributed set of keypoints. Afterwards, every keypoint is characterized by a scale, rotation and translation invariant 3D surface descriptor, called the ‘radial geodesic distance-slope histogram’. Similar keypoints descriptors on the source and target datasets are then matched using bipartite graph matching, followed by a modified-RANSAC for outlier removal. The second proposed method is based on 2D keypoint matching performed on height map images of the 3D point clouds. Height map images are generated by projecting the 3D point clouds onto a planimetric plane. Afterwards, a multi-scale wavelet 2D keypoint detector with ANMS is proposed to extract keypoints on the height maps. Then, a scale, rotation and translation-invariant 2D descriptor referred to as the ‘Gabor, Log-Polar- Rapid Transform’ descriptor is computed for all keypoints. Finally, source and target height map keypoint correspondences are determined using a bi-directional nearest neighbour matching, together with the modified-RANSAC for outlier removal. iii Each method is assessed on multi-sensor, urban and non-urban 3D point cloud datasets. Results show that unlike the 3D-based method, the height map-based approach is able to align source and target datasets with differences in point density, point distribution and missing point data. Findings also show that the 3D-based method obtained lower transformation errors and a greater number of correspondences when the source and target have similar point characteristics. The 3D-based approach attained absolute mean alignment differences in the range of 0.23m to 2.81m, whereas the height map approach had a range from 0.17m to 1.21m. These differences meet the proximity requirements of the data characteristics and the further application of fine co-registration approaches. iv Acknowledgements I would like to express the utmost thanks and appreciation to my PhD. supervisor, Dr. Costas Armenakis for his constant support, guidance, patience and mentorship during my time at York University. Under his numerous years of tutelage, I have learnt many things about academics and about life in general. I cannot thank him enough for the opportunity to do so. I’d also like to sincerely thank Dr. Gunho Sohn for his years of support. He and Dr. Armenakis graciously welcomed me into the GeoICT Lab when I first came to Canada. I have thoroughly enjoyed working with and learning from him. I also wish to extend my thanks and appreciation to Dr. Regina Lee, Dr. Burton Ma, Dr. Derek Lichti and Dr. Franz Newland for their valuable time and effort on reviewing this dissertation. I also thank my colleagues at the GEOICT lab with whom I have worked on several interesting research projects over the years. I would like to sincerely thank Dr. James Elder, Department of Electrical Engineering and Computer Science, York University, for providing the Aeryon Scout UAV video data, and Mike Demuth (Geological Survey of Canada) and Alexander Chichagov (Canada Centre for Mapping and Earth Observation), both from Natural Resources Canada (NRCan) for providing the Columbia Icefield datasets. Teledyne Optech and First Base Solutions are much thanked for providing the urban datasets. Finally, I dedicate this work to my parents, Sundar and Deokie Persad and thank them for their endless support. v Table of Contents Abstract ii Acknowledgements iv Table of Contents v List of Tables ix List of Figures xi List of Acronyms xv 1 Introduction 1 1.1 Initial alignment versus refined alignment .................................................................... 4 1.2 Initial alignment: global versus local methods .............................................................. 7 1.3 Overview and objectives ................................................................................................... 8 1.4 Contributions .................................................................................................................... 10 1.5 Organization ..................................................................................................................... 11 2 Related Works on Initial Point Cloud Alignment 14 2.1 3D Descriptor-based methods ........................................................................................ 15 2.1.1 3D keypoint extraction ........................................................................................ 15 2.1.2 Matching of 3D keypoints using descriptors ................................................... 18 2.2 3D Non-descriptor-based methods ................................................................................ 24 vi 2.3 2D image-based methods ................................................................................................ 27 2.4 Summary ........................................................................................................................... 29 3 A 3D-based Approach for Point Cloud Alignment 31 3.1 3D-based Point Cloud Alignment Methodology ........................................................ 32 3.2 Extraction of 3D Surface Keypoints ............................................................................ 34 3.2.1 Scale invariance for 3D keypoints ..................................................................... 37 3.2.2 Keypoint refinement by adaptive non-maxima suppression .......................... 41 3.3 3D Surface Descriptors for Keypoints ......................................................................... 44 3.3.1 Rigid invariance for local 3D descriptors ......................................................... 44 3.3.2 Local 3D surface description .............................................................................. 46 3.3.3 3D Keypoint matching using RGSH descriptor .............................................. 50 3.3.4 Removal of 3D keypoint correspondence outliers .......................................... 51 3.4 Summary .......................................................................................................................... 54 4 A Height Map-based Approach for Point Cloud Alignment 55 4.1 Height Map-based Point Cloud Alignment Methodology ........................................ 56 4.2 Multi-scale 2D keypoint extraction .............................................................................. 58 4.2.1 2D keypoint extraction using DTCWT ............................................................. 60 4.3 Scale, rotation and translation invariant 2D keypoint descriptor ............................. 66 4.3.1 Log-polar sampling and mapping for 2D scale and rotation invariance ....... 68 4.3.1.1 Generation of Gabor filter-based derivatives ...................................... 71 vii 4.3.2 Descriptor invariance to 2D cyclic-shifts using the Rapid Transform .......... 73 4.3.3 2D keypoint matching using GLP-RT descriptor ............................................ 77 4.4 Summary ......................................................................................................................... 79 5 Results and Analysis 80 5.1 Results for Method 1: 3D-based Point Cloud Alignment .......................................... 81 5.1.1 Empirical selection of RGSH descriptor bin size ............................................. 82 5.1.2 Case 1: Same sensor datasets, different coordinate systems .......................... 88 5.1.3 Case 2: Different sensor datasets, different coordinate systems .................... 94 5.2 Results for Method 2: Height Map-based Point Cloud Alignment ......................... 102 5.2.1 Experimental datasets ........................................................................................ 102 5.2.1.1 Dataset 1 (Urban, Loc1) ....................................................................... 103 5.2.1.2 Dataset 2 (Urban, Loc2) ....................................................................... 104 5.2.1.3 Dataset 3 (Non-Urban, Loc3) .............................................................. 106 5.2.1.4 Tuning and testing datasets .................................................................. 107 5.2.2 Empirical tuning: Selection of GLP-RT descriptor parameters ................... 108 5.2.2.1 The minimum radius ............................................................................. 110 5.2.2.2 The maximum radius ............................................................................ 111 5.2.2.3 The number of rays and number of rings ........................................... 112 5.2.3 Testing experiment: Assessment of the 2D height map approach with other 2D keypoint detectors and descriptors ............................................................ 114 5.2.4 Accuracy analysis of 2D height map-based point cloud co-registration ..... 120 viii 5.3 Overall assessment of the proposed 3D-based and height map-based co- registration methods ...................................................................................................... 128 5.3.1 Observations for real datasets 1 and 2 (Urban, Loc1 and Urban, Loc2) .... 129 5.3.2 Observations for real dataset 3 (Non-Urban, Loc3) ...................................... 132 5.4 Computation time .......................................................................................................... 134 6 Conclusions 136 6.1 Research outcomes ........................................................................................................ 137 6.1.1 Summary of the 3D-based point cloud alignment method............................ 137 6.1.2 Summary of the Height map-based point cloud alignment method ............ 139 6.2 Recommendations for future work ............................................................................. 140 References 143 Appendix A : Bipartite matching using the Hungarian method 155 Appendix B : Rapid Transform 158 ix List of Tables 5.1 Manually-defined transformation parameters used for generating target point clouds of the 4 training sites in the tuning dataset ..................................................... 85 5.2 Descriptor matching for various keypoints on Figure 5.4 ........................................ 92 5.3 Co-registration result for ‘Case 1’ Urban dataset ...................................................... 93 5.4 Co-registration result for ‘Case 1’ Icefield (Non-Urban) dataset ............................ 93 5.5 Average Angular and Translation errors for ‘Case 1’ datasets ................................ 94 5.6 Co-registration result for ‘Case 2’ Urban dataset ...................................................... 96 5.7 Co-registration result for ‘Case 2’ Icefield (Non-Urban) dataset ............................ 96 5.8 Average Angular and Translation errors for ‘Case 2’ datasets ................................ 98 5.9 Simulated and real source and target datasets which are used for the empirical tuning ............................................................................................................................. 109 5.10 Simulated and real source and target datasets which are used for the testing experiment .................................................................................................................... 109 5.11 Optimal GLP-RT descriptor parameters after tuning ............................................ 112 5.12 Combinations of 2D keypoint detectors and 2D descriptors evaluated on the height map testing datasets ....................................................................................... 116 5.13 Co-registration result for real test dataset 1 (Urban, Loc1) ................................. 122 5.14 Co-registration result for real test dataset 2 (Urban, Loc2) ................................. 122 5.15 Co-registration result for real test dataset 3 (Non-Urban, Loc3) ........................ 123 5.16 Co-registration errors using proposed multi-scale wavelet 2D keypoint detector x and GLP-RT descriptor .............................................................................................. 131 5.17 Co-registration errors using proposed surface curvature-based 3D detector and RGSH descriptor ......................................................................................................... 131 5.18 Co-registration errors using 3D-SIFT 3D keypoint detector and FPFH descriptor ................................................................................................. 131 5.19 Co-registration errors using 3D-SIFT 3D keypoint detector and SHOT descriptor ................................................................................................. 132 5.20 Comparison of 3D keypoint detectors for ‘real dataset 3’ based on localization accuracy and similarity of local keypoint scales.................................................... 134