TORUS 2 – Toward an Open Resource Using Services TORUS 2 – Toward an Open Resource Using Services Cloud Computing for Environmental Data Edited by Dominique Laffly First published 2020 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc. Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms and licenses issued by the CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the undermentioned address: ISTE Ltd John Wiley & Sons, Inc. 27-37 St George’s Road 111 River Street London SW19 4EU Hoboken, NJ 07030 UK USA www.iste.co.uk www.wiley.com © ISTE Ltd 2020 The rights of Dominique Laffly to be identified as the author of this work have been asserted by him in accordance with the Copyright, Designs and Patents Act 1988. Library of Congress Control Number: 2019956836 British Library Cataloguing-in-Publication Data A CIP record for this book is available from the British Library ISBN 978-1-78630-600-5 Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Part 1. Earth Science Remote Sensing . . . . . . . . . . . . . . . . . . . . . . . xvii Introduction to Part 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix Dominique LAFFLY Chapter 1. A Brief History of Remote Sensing . . . . . . . . . . . . . . . . . . . 1 Dominique LAFFLY 1.1. History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2. Fields of application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.3. Orbits, launchers and platforms . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.4. The acquired data are digital images . . . . . . . . . . . . . . . . . . . . . . . . 12 1.5. So what is remote sensing? Some definitions . . . . . . . . . . . . . . . . . . . 14 1.6. Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 1.7. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Chapter 2. Physics of RS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Luca TOMASSETTI 2.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.2. Remote sensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.3. Fundamental properties of electromagnetic waves . . . . . . . . . . . . . . . . 29 2.3.1. Wave equation and solution . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.3.2. Quantum properties of electromagnetic radiation . . . . . . . . . . . . . . 30 2.3.3. Polarization, coherence, group and phase velocity, the Doppler effect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 vi TORUS 2 – Toward an Open Resource Using Services 2.4. Radiation quantities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 2.4.1. Spectral quantities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 2.4.2. Luminous quantities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 2.5. Generation of electromagnetic waves . . . . . . . . . . . . . . . . . . . . . . . 34 2.6. Detection of electromagnetic waves . . . . . . . . . . . . . . . . . . . . . . . . 37 2.7. Interaction of electromagnetic waves with matter . . . . . . . . . . . . . . . . 38 2.7.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 2.7.2. Interaction mechanisms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.8. Solid surfaces sensing in the visible and near infrared . . . . . . . . . . . . . . 41 2.8.1. Wave-surface interaction mechanisms . . . . . . . . . . . . . . . . . . . . 43 2.9. Radiometric and geometric resolutions . . . . . . . . . . . . . . . . . . . . . . 45 2.10. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Chapter 3. Image Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Dominique LAFFLY 3.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.2. Image quality – geometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.2.1. Whiskbroom concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 3.2.2. Pushbroom concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 3.2.3. Full frame concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 3.2.4. Optical geometric distortions . . . . . . . . . . . . . . . . . . . . . . . . . 64 3.2.5. Relief distortions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 3.2.6. Inverse location model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 3.2.7. Direct location model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 3.2.8. Root Mean Square (RMS) validation . . . . . . . . . . . . . . . . . . . . . 72 3.2.9. Resampling methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 3.2.10. Image geometric quality to assume geographical space continuity . . . . 75 3.3. Image quality – radiometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 3.3.1. Radiometric model of the instrument . . . . . . . . . . . . . . . . . . . . . 78 3.3.2. Radiometric equalization and calibration . . . . . . . . . . . . . . . . . . . 79 3.3.3. Radiometric signal noise reduction (SNR) . . . . . . . . . . . . . . . . . 81 3.3.4. Radiometric physical value . . . . . . . . . . . . . . . . . . . . . . . . . . 82 3.3.5. Image quality – resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 3.4. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 3.5. Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 3.6. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 Chapter 4. Remote Sensing Products . . . . . . . . . . . . . . . . . . . . . . . . 95 Van Ha PHAM, Viet Hung LUU, Anh PHAN, Dominique LAFFLY, Quang Hung BUI and Thi Nhat Thanh NGUYEN 4.1. Atmospheric observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 4.1.1. Introduction to common atmospheric gases and particles . . . . . . . . . . 95 Contents vii 4.1.2. Introduction to meteorological parameters . . . . . . . . . . . . . . . . . . 103 4.1.3. Atmospheric observation from satellite . . . . . . . . . . . . . . . . . . . . 107 4.2. Land observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 4.2.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 4.2.2. Land cover/land use classification system . . . . . . . . . . . . . . . . . . 129 4.2.3. Legend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 4.2.4. Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 4.2.5. Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 4.2.6. Global land cover datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 4.3. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 4.4. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 Chapter 5. Image Processing in Spark . . . . . . . . . . . . . . . . . . . . . . . . 163 Yannick LE NIR, Florent DEVIN, Thomas BALDAQUIN, Pierre MESLER LAZENNEC, Ji Young JUNG, Se-Eun KIM, Hyeyoung KWOON, Lennart NILSEN, Yoo Kyung LEE and Dominique LAFFLY 5.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 5.2. Prediction map generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 5.2.1. Spark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 5.2.2. Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 5.2.3. Naive method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 5.2.4. Advanced method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 5.3. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 Chapter 6. Satellite Image Processing using Spark on the HUPI Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 Vincent MORENO and Minh Tu NGUYEN 6.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 6.2. Presentation of GeoTrellis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 6.3. Using GeoTrellis in Hupi-Notebook . . . . . . . . . . . . . . . . . . . . . . . . 174 6.3.1. Some core concepts of GeoTrellis . . . . . . . . . . . . . . . . . . . . . . . 177 6.3.2. Computation of NDVI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 6.3.3. Compare two NDVI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 6.3.4. Descriptive statistics of NDVI per Tile . . . . . . . . . . . . . . . . . . . . 178 6.3.5. K-means . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 6.4. Workflows in HDFS: automatize image processing . . . . . . . . . . . . . . . 181 6.4.1. Create a jar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 6.4.2. Monitor the Spark jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 6.4.3. Tune performance of the Spark job . . . . . . . . . . . . . . . . . . . . . . 183 6.4.4. Create a workflow in Hupi-Studio . . . . . . . . . . . . . . . . . . . . . . . 184 viii TORUS 2 – Toward an Open Resource Using Services 6.5. Visualizations in Hupi-Front . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 6.6. Cloud service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 6.7. Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 Chapter 7. Remote Sensing Case Studies . . . . . . . . . . . . . . . . . . . . . 191 Van Ha PHAM, Thi Nhat Thanh NGUYEN and Dominique LAFFLY 7.1. Satellite AOD validation using R . . . . . . . . . . . . . . . . . . . . . . . . . . 191 7.1.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 7.1.2. Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 7.1.3. Validation methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 7.1.4. Experiments and results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 7.1.5. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204 7.2. Georeferencing satellite images . . . . . . . . . . . . . . . . . . . . . . . . . . 204 7.2.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204 7.2.2. Georeferencing methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 7.2.3. Datasets and methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 7.2.4. Results and discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 7.3. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216 7.4. Appendix: R source code of validation process . . . . . . . . . . . . . . . . . . 217 7.5. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 Conclusion to Part 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 Dominique LAFFLY Part 2. GIS Application and Geospatial Data Infrastructure . . . . . . . . . . 227 Chapter 8. Overview of GIS Application . . . . . . . . . . . . . . . . . . . . . . . 229 Quang Huy MAN 8.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 8.2. Enterprise GIS for environmental management . . . . . . . . . . . . . . . . . . 230 8.3. GIS and decision-making in planning and management . . . . . . . . . . . . . 232 8.3.1. Data quality and control . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 8.3.2. Decision support systems (DSS) . . . . . . . . . . . . . . . . . . . . . . . 233 8.3.3. Integrating GIS with the DSS . . . . . . . . . . . . . . . . . . . . . . . . . 234 8.4. GIS for water-quality management . . . . . . . . . . . . . . . . . . . . . . . . . 235 8.5. GIS for land-use planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236 8.6. Application of the technology in LUP and management . . . . . . . . . . . . . 240 8.6.1. Computers and software programs applied to LUP and management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 8.6.2. Application of GIS analysis and MCE in land-use planning and management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 8.7. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 Contents ix Chapter 9. Spatial Data Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . 247 Quang Hung BUI, Quang Thang LUU, Duc Van HA, Tuan Dung PHAM, Sanya PRASEUTH and Dominique LAFFLY 9.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 9.2. Spatial data infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 9.3. Components of spatial data infrastructure . . . . . . . . . . . . . . . . . . . . . 249 9.4. Open standards for spatial data infrastructure . . . . . . . . . . . . . . . . . . . 251 9.4.1. Open geospatial consortium (OGC) . . . . . . . . . . . . . . . . . . . . . 251 9.4.2. OGC’s open standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 9.4.3. Usage of OGC’s open standards in SDI . . . . . . . . . . . . . . . . . . . . 255 9.5. Server architecture models for the National Spatial Data Infrastructure and Geospatial One-Stop (GOS) portal . . . . . . . . . . . . . . . . . 256 9.5.1. GOS portal architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 9.5.2. Standards for GOS portal architecture . . . . . . . . . . . . . . . . . . . . 257 9.5.3. Taxonomy of geospatial server architecture . . . . . . . . . . . . . . . . . 257 9.5.4. Three reference architectures for server architecture model . . . . . . . . . 258 9.6. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260 List of Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 Summaries of other volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 Preface Why TORUS? Toward an Open Resource Using Services, or How to Bring Environmental Science Closer to Cloud Computing Geography, Ecology, Urbanism, Geology and Climatology – in short, all environmental disciplines are inspired by the great paradigms of Science: they were first descriptive before evolving toward systemic and complexity. The methods followed the same evolution, from the inductive of the initial observations one approached the deductive of models of prediction based on learning. For example, the Bayesian is the preferred approach in this book (see Volume 1, Chapter 5), but random trees, neural networks, classifications and data reductions could all be developed. In the end, all the methods of artificial intelligence (IA) are ubiquitous today in the era of Big Data. We are not unaware, however, that, forged in Dartmouth in 1956 by John McCarthy, Marvin Minsky, Nathaniel Rochester and Claude Shannon, the term artificial intelligence is, after a long period of neglect at the heart of the future issues of the exploitation of massive data (just like the functional and logical languages that accompanied the theory: LISP, 1958, PROLOG, 1977 and SCALA, today – see Chapter 8). All the environmental disciplines are confronted with this reality of massive data, with the rule of the 3+2Vs: Volume, Speed (from the French translation, “Vitesse”), Variety, Veracity, Value. Every five days – or even less – and only for the optical remote sensing data of the Sentinel 2a and 2b satellites, do we have a complete coverage of the Earth at a spatial resolution of 10 m for a dozen wavelengths. How do we integrate all this, how do we rethink the environmental disciplines where we must now consider at the pixel scale (10 m) an overall analysis of 510 million km2 or more than 5 billion pixels of which there are 1.53 billion for land only? And more important in fact, how do we validate automatic processes and accuracy of results?