Cloud Computing in Remote Sensing Cloud Computing in Remote Sensing Lizhe Wang Jining Yan Yan Ma CRCPress Taylor&FrancisGroup 6000BrokenSoundParkwayNW,Suite300 BocaRaton,FL33487-2742 ©2020byTaylor&FrancisGroup,LLC CRCPressisanimprintofTaylor&FrancisGroup,anInformabusiness NoclaimtooriginalU.S.Governmentworks Printedonacid-freepaper InternationalStandardBookNumber-13:978-1-138-59456-2(Hardback) Thisbookcontainsinformationobtainedfromauthenticandhighlyregardedsources.Rea- sonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the conse- quences of their use. The authors and publishers have attempted to trace the copyright holdersofallmaterialreproducedinthispublicationandapologizetocopyrightholdersif permissiontopublishinthisformhasnotbeenobtained.Ifanycopyrightmaterialhasnot beenacknowledgedpleasewriteandletusknowsowemayrectifyinanyfuturereprint. Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means,nowknownorhereafterinvented,includingphotocopying,microfilming,andrecord- ing,orinanyinformationstorageorretrievalsystem,withoutwrittenpermissionfromthe publishers. For permission to photocopy or use material electronically from this work, please access www.copyright.com(http://www.copyright.com/)orcontacttheCopyrightClearanceCen- ter, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not- for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system ofpaymenthasbeenarranged. Trademark Notice:Productorcorporatenamesmaybetrademarksorregisteredtrade- marks,andareusedonlyforidentificationandexplanationwithoutintenttoinfringe. Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com and the CRC Press Web site at http://www.crcpress.com Contents Preface xi 1 Remote Sensing and Cloud Computing 1 1.1 Remote Sensing . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.1 Remote sensing definition . . . . . . . . . . . . . . . . . 1 1.1.2 Remote sensing big data . . . . . . . . . . . . . . . . . 2 1.1.3 Applications of remote sensing big data . . . . . . . . 3 1.1.4 Challenges of remote sensing big data . . . . . . . . . 5 1.1.4.1 Data integration challenges . . . . . . . . . . 5 1.1.4.2 Data processing challenges . . . . . . . . . . 5 1.2 Cloud Computing . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2.1 Cloud service models . . . . . . . . . . . . . . . . . . . 6 1.2.2 Cloud deployment models . . . . . . . . . . . . . . . . . 7 1.2.3 Security in the Cloud . . . . . . . . . . . . . . . . . . . 7 1.2.4 Open-source Cloud frameworks . . . . . . . . . . . . . 8 1.2.4.1 OpenStack . . . . . . . . . . . . . . . . . . . 8 1.2.4.2 Apache CloudStack . . . . . . . . . . . . . . 10 1.2.4.3 OpenNebula . . . . . . . . . . . . . . . . . . 10 1.2.5 Big data in the Cloud . . . . . . . . . . . . . . . . . . 12 1.2.5.1 Big data management in the Cloud . . . . . 12 1.2.5.2 Big data analytics in the Cloud . . . . . . . 12 1.3 Cloud Computing in Remote Sensing . . . . . . . . . . . . . 14 2 Remote Sensing Data Integration in a Cloud Computing Environment 17 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.2 Background on Architectures for Remote Sensing Data Integra- tion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.2.1 Distributed integration of remote sensing data . . . . 18 2.2.2 OODT: a data integration framework . . . . . . . . . 19 2.3 Distributed Integration of Multi-Source Remote Sensing Data 20 2.3.1 The ISO 19115-based metadata transformation . . . . 20 2.3.2 Distributed multi-source remote sensing data integration 22 v vi Contents 2.4 Experiment and Analysis . . . . . . . . . . . . . . . . . . . . 24 2.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3 Remote Sensing Data Organization and Management in a Cloud Computing Environment 29 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.2 Preliminaries and Related Techniques . . . . . . . . . . . . . . 31 3.2.1 Spatial organization of remote sensing data . . . . . . . 31 3.2.2 MapReduce and Hadoop . . . . . . . . . . . . . . . . . 32 3.2.3 HBase . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.2.4 Elasticsearch . . . . . . . . . . . . . . . . . . . . . . . 33 3.3 LSI Organization Model of Multi-Source Remote Sensing Data 35 3.4 Remote Sensing Big Data Management in a Parallel File System 38 3.4.1 Full-text index of multi-source remote sensing metadata 38 3.4.2 Distributed data retrieval . . . . . . . . . . . . . . . . 40 3.5 Remote Sensing Big Data Management in the Hadoop Ecosys- tem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.5.1 Data organization and storage component . . . . . . . 42 3.5.2 Data index and search component . . . . . . . . . . . 43 3.6 Metadata Retrieval Experiments in a Parallel File System . . 45 3.6.1 LSI model-based metadata retrieval experiments in a parallel file system . . . . . . . . . . . . . . . . . . . . 45 3.6.2 Comparative experiments and analysis . . . . . . . . . 48 3.6.2.1 Comparative experiments . . . . . . . . . . . 48 3.6.2.2 Results analysis . . . . . . . . . . . . . . . . 49 3.7 Metadata Retrieval Experiments in the Hadoop Ecosystem . . 51 3.7.1 Time comparisons of storing metadata in HBase . . . 52 3.7.2 Time comparisons of loading metadata from HBase to Elasticsearch . . . . . . . . . . . . . . . . . . . . . . . 52 3.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4 High Performance Remote Sensing Data Processing in a Cloud Computing Environment 55 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 4.2 High Performance Computing for RS Big Data: State of the Art 58 4.2.1 Cluster computing for RS data processing . . . . . . . 58 4.2.2 Cloud computing for RS data processing . . . . . . . . 59 4.2.2.1 Programming models for big data . . . . . . 60 4.2.2.2 Resource management and provisioning . . . 60 4.3 Requirements and Challenges: RSCloud for RS Big Data . . . 61 4.4 pipsCloud: High Performance Remote Sensing Clouds . . . . 62 4.4.1 The system architecture of pipsCloud . . . . . . . . . 63 4.4.2 RS data management and sharing . . . . . . . . . . . 65 Contents vii 4.4.2.1 HPGFS: distributed RS data storage with application-aware data layouts and copies . . . 67 4.4.2.2 RS metadata management with NoSQL database . . . . . . . . . . . . . . . . . . . . 68 4.4.2.3 RS data index with Hilbert R+tree . . . . . 69 4.4.2.4 RS data subscription and distribution . . . . . 71 4.4.3 VE-RS: RS-specific HPC environment as a service . . 72 4.4.3.1 On-demand HPC cluster platforms with bare-metal provisioning . . . . . . . . . . . . 73 4.4.3.2 Skeletal programming for RS big data process- ing. . . . . . . . . . . . . . . . . . . . . . . . 76 4.4.4 VS-RS: Cloud-enabled RS data processing system . . . 77 4.4.4.1 Dynamic workflow processing for RS applica- tions in the Cloud . . . . . . . . . . . . . . . 78 4.5 Experiments and Discussion . . . . . . . . . . . . . . . . . . 82 4.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 5 Programming Technologies for High Performance Remote SensingDataProcessinginaCloudComputingEnvironment 89 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 5.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 5.3 Problem Definition . . . . . . . . . . . . . . . . . . . . . . . 92 5.3.1 Massive RS data . . . . . . . . . . . . . . . . . . . . . 92 5.3.2 Parallel programmability . . . . . . . . . . . . . . . . 93 5.3.3 Data processing speed . . . . . . . . . . . . . . . . . . 94 5.4 Design and Implementation . . . . . . . . . . . . . . . . . . . 94 5.4.1 Generic algorithm skeletons for remote sensing applica- tions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 5.4.1.1 Categories of remote sensing algorithms . . . 98 5.4.1.2 Generic RS farm-pipeline skeleton . . . . . . 98 5.4.1.3 Generic RS image-wrapper skeleton . . . . . 102 5.4.1.4 Generic feature abstract skeleton . . . . . . . 105 5.4.2 Distributed RS data templates . . . . . . . . . . . . . 108 5.4.2.1 RSData templates . . . . . . . . . . . . . . . 108 5.4.2.2 Dist RSData templates . . . . . . . . . . . . . 111 5.5 Experiments and Discussion . . . . . . . . . . . . . . . . . . 115 5.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 6 Construction and Management of Remote Sensing Produc- tion Infrastructures across Multiple Satellite Data Centers 121 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 6.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 123 6.3 Infrastructures Overview . . . . . . . . . . . . . . . . . . . . 124 viii Contents 6.3.1 Target environment . . . . . . . . . . . . . . . . . . . 124 6.3.2 MDCPS infrastructures overview . . . . . . . . . . . . 125 6.4 Design and Implementation . . . . . . . . . . . . . . . . . . . 128 6.4.1 Data management . . . . . . . . . . . . . . . . . . . . 128 6.4.1.1 Spatial metadata management for co-processing . . . . . . . . . . . . . . . . . . 130 6.4.1.2 Distributed file management . . . . . . . . . . 131 6.4.2 Workflow management . . . . . . . . . . . . . . . . . . 133 6.4.2.1 Workflow construction . . . . . . . . . . . . . 136 6.4.2.2 Task scheduling . . . . . . . . . . . . . . . . . 137 6.4.2.3 Workflow fault-tolerance . . . . . . . . . . . . 141 6.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 6.5.1 Related experiments on dynamic data management . . 142 6.5.2 Related experiments on workflow management . . . . 146 6.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 6.6.1 System architecture . . . . . . . . . . . . . . . . . . . . 147 6.6.2 System feasibility . . . . . . . . . . . . . . . . . . . . . 148 6.6.3 System scalability . . . . . . . . . . . . . . . . . . . . 148 6.7 Conclusions and Future Work . . . . . . . . . . . . . . . . . 148 7 Remote Sensing Product Production in an OpenStack-Based Cloud Computing Environment 151 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 7.2 Background and Related Work . . . . . . . . . . . . . . . . . 153 7.2.1 Remote sensing products . . . . . . . . . . . . . . . . 153 7.2.1.1 Fine processing products . . . . . . . . . . . 154 7.2.1.2 Inversion index products . . . . . . . . . . . 154 7.2.1.3 Thematic products. . . . . . . . . . . . . . . 154 7.2.2 Remote sensing production system . . . . . . . . . . . 155 7.3 Cloud-Based Remote Sensing Production System . . . . . . 156 7.3.1 Program framework . . . . . . . . . . . . . . . . . . . 156 7.3.2 System architecture . . . . . . . . . . . . . . . . . . . . 157 7.3.3 Knowledge base and inference rules. . . . . . . . . . . 159 7.3.3.1 The upper and lower hierarchical relationship database . . . . . . . . . . . . . . . . . . . . 159 7.3.3.2 Input/output database of every kind of remote sensing product . . . . . . . . . . . . . . . . 160 7.3.3.3 Inference rules for production demand data selection . . . . . . . . . . . . . . . . . . . . . 161 7.3.3.4 Inference rules for workflow organization . . . 161 7.3.4 Business logic . . . . . . . . . . . . . . . . . . . . . . . 162 7.3.5 Active service patterns . . . . . . . . . . . . . . . . . . 165 7.4 Experiment and Case Study . . . . . . . . . . . . . . . . . . . 167 7.4.1 Global scale remote sensing production . . . . . . . . . 167 Contents ix 7.4.2 Regional scale mosaic production . . . . . . . . . . . . 168 7.4.3 Local scale change detection. . . . . . . . . . . . . . . 170 7.4.3.1 Remote sensing data cube. . . . . . . . . . . . 171 7.4.3.2 Local scale time-series production . . . . . . . 171 7.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 8 KnowledgeDiscoveryandInformationAnalysisfromRemote Sensing Big Data 175 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 8.2 Preliminaries and Related Work . . . . . . . . . . . . . . . . 176 8.2.1 Knowledge discovery categories . . . . . . . . . . . . . 176 8.2.2 Knowledge discovery methods . . . . . . . . . . . . . . 178 8.2.3 Related work . . . . . . . . . . . . . . . . . . . . . . . 179 8.3 Architecture Overview . . . . . . . . . . . . . . . . . . . . . 180 8.3.1 Target data and environment . . . . . . . . . . . . . . 180 8.3.2 FRSDC architecture overview . . . . . . . . . . . . . . . 181 8.4 Design and Implementation . . . . . . . . . . . . . . . . . . . 182 8.4.1 Feature data cube . . . . . . . . . . . . . . . . . . . . 182 8.4.1.1 Spatial feature object in FRSDC . . . . . . . 182 8.4.1.2 Data management . . . . . . . . . . . . . . . 182 8.4.2 Distributed executed engine . . . . . . . . . . . . . . . 184 8.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 8.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 9 AutomaticConstructionofCloudComputingInfrastructures in Remote Sensing 191 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 9.2 Definition of the Remote Sensing Oriented Cloud Computing Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . 192 9.2.1 Generally used cloud computing infrastructure . . . . 193 9.2.2 Remote sensing theme oriented cloud computing infras- tructure . . . . . . . . . . . . . . . . . . . . . . . . . . 193 9.3 Design and Implementation of Remote Sensing Oriented Cloud Computing Infrastructure . . . . . . . . . . . . . . . . . . . . 195 9.3.1 System architecture design . . . . . . . . . . . . . . . 195 9.3.2 System workflow design . . . . . . . . . . . . . . . . . 196 9.3.3 System module design . . . . . . . . . . . . . . . . . . 198 9.4 Key Technologies of Remote Sensing Oriented Cloud Infrastruc- ture Automatic Construction . . . . . . . . . . . . . . . . . . 200 9.4.1 Automatic deployment based on OpenStack and Salt- Stack . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 9.4.2 Resource monitoring based on Ganglia . . . . . . . . . 203 9.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205