This book provides a global review of optical satellite image and data compression theories, algorithms, and system implementations. Consisting of nine chapters, it describes a variety of lossless and near-lossless data-compression techniques and three international satellite-data-compression standards. The author shares his firsthand experience and research results in developing novel satellite-data-compression techniques for both onboard and on-ground use, user assessments of the impact that data compression has on satellite data applications, building hardware compression systems, and optimizing and deploying systems. Written with both postgraduate students and advanced professionals in mind, this handbook addresses important issues of satellite data compression and implementation, and it presents an end-to-end treatment of data compression technology. WWant to learn more about the calibration and eenhancement of spaceborne optical sensors and mmethods for image enhancement and fusion? SPIE PPress presents this book's companion text, Optical SSatellite Signal Processing and Enhancement, also wwritten by Shen-En Qian. QIAN P.O. Box 10 Bellingham, WA 98227-0010 ISBN: 9780819497871 SPIE Vol. No.: PM241 Optical Satellite Data Compression and Implementation Shen-En Qian SPIE PRESS Bellingham, Washington USA Library of Congress Cataloging-in-Publication Data Qian, Shen-En. Optical satellite data compression and implementation / Shen-En Qian. pages cm Includes bibliographical references and index. ISBN 978-0-8194-9787-1 1. Datacompression(Computerscience). 2. Imagingsystems Imagequality. 3. Signalprocessing. 4. Codingtheory. 5. Opticalimages. I. Title. QA76.9.D33 2013 629.43'7 dc23 2013944363 Published by SPIE The International Society for Optical Engineering P.O. Box 10 Bellingham, Washington 98227-0010 USA Phone: +1 360 676 3290 Fax: +1 360 647 1445 Email: [email protected] Web: http://spie.org Copyright © 2013 Society of Photo-Optical Instrumentation Engineers (SPIE) All rights reserved. No part of this publication may be reproduced or distributed in any form or by any means without written permission of the publisher. Thecontentofthisbookreflectstheworkandthoughtoftheauthor(s).Everyefforthas beenmadetopublishreliableandaccurateinformationherein,butthepublisherisnot responsible for the validity of the information or for any outcomes resulting from reliance thereon. Printed in the United States of America. First printing Contents Preface xiii List of Terms and Acronyms xvii 1 Needs for Data Compression and Image Quality Metrics 1 1.1 Needs for Satellite Data Compression 1 1.2 Quality Metrics of Satellite Images 4 1.3 Full-Reference Metrics 5 1.3.1 Conventional full-reference metrics 6 1.3.1.1 Mean-square error (MSE) 6 1.3.1.2 Relative-mean-square error (ReMSE) 7 1.3.1.3 Signal-to-noise ratio (SNR) 7 1.3.1.4 Peak signal-to-noise ratio (PSNR) 7 1.3.1.5 Maximum absolute difference (MAD) 7 1.3.1.6 Percentage maximum absolute difference (PMAD) 8 1.3.1.7 Mean absolute error (MAE) 8 1.3.1.8 Correlation coefficient (CC) 8 1.3.1.9 Mean-square spectral error (MSSE) 9 1.3.1.10 Spectral correlation (SC) 9 1.3.1.11 Spectral angle (SA) 9 1.3.1.12 Maximum spectral information divergence (MSID) 10 1.3.1.13 ERGAS for multispectral image after pan-sharpening 10 1.3.2 Perceived-visual-quality-based full-reference metrics 11 1.3.2.1 Universal image-quality index 11 1.3.2.2 Multispectral image-quality index 12 1.3.2.3 Quality index for multi- or hyperspectral images 14 1.3.2.4 Structural similarity index 15 1.3.2.5 Visual information fidelity 17 1.4 Reduced-Reference Metrics 18 1.4.1 Four RR metrics for spatial-resolution-enhanced images 20 1.4.2 RR metric using the wavelet-domain natural-image statistic model 22 1.5 No-Reference Metrics 24 1.5.1 Statistic-based methods 24 v vi Contents 1.5.1.1 Entropy 24 1.5.1.2 Energy compaction 25 1.5.1.3 Coding gain 25 1.5.2 NR metric for compressed images using JPEG 26 1.5.3 NR metric for pan-sharpened multispectral image 27 1.5.3.1 Spectral distortion index 28 1.5.3.2 Spatial distortion index 29 1.5.3.3 Jointly spectral and spatial quality index 29 References 29 2 Lossless Satellite Data Compression 33 2.1 Introduction 33 2.2 Review of Lossless Satellite Data Compression 35 2.2.1 Prediction-based methods 35 2.2.2 Transform-based methods 38 2.3 Entropy Encoders 40 2.3.1 Adaptive arithmetic coding 40 2.3.2 Golomb coding 41 2.3.3 Exponential-Golomb coding 42 2.3.4 Golomb power-of-two coding 42 2.4 Predictors for Hyperspectral Datacubes 44 2.4.1 1D nearest-neighboring predictor 45 2.4.2 2D/3D predictors 45 2.4.3 Predictors within a focal plane image 45 2.4.4 Adaptive selection of predictor 47 2.4.5 Experimental results of the predictors 48 2.4.5.1 Compressionresultsusingfixedcoefficientpredictors 49 2.4.5.2 Compression results using variable coefficient predictors 50 2.4.5.3 Compression results using adaptive selection of predictor 51 2.5 Lookup-Table-Based Prediction Methods 53 2.5.1 Single-lookup-table prediction 53 2.5.2 Locally averaged, interband-scaling LUT prediction 54 2.5.3 Quantized-index LUT prediction 56 2.5.4 Multiband LUT prediction 56 2.6 Vector-Quantization-Based Prediction Methods 57 2.6.1 Linear prediction 57 2.6.2 Grouping based on bit-length 58 2.6.3 Vector quantization with precomputed codebooks 58 2.6.4 Optimal bit allocation 59 2.6.5 Entropy coding 59 2.7 Band Reordering 60 2.8 Transform-Based Lossless Compression Using the KLT and DCT 61 Contents vii 2.9 Wavelet-Transform-Based Methods 62 2.9.1 Wavelet decomposition structure 63 2.9.2 Lossy-to-lossless compression: 3D set-partitioning embedded block 63 2.9.3 Lossy-to-lossless compression: 3D embedded zeroblock coding 66 References 68 3 International Standards for Spacecraft Data Compression 75 3.1 CCSDS and Three Data Compression Standards 75 3.2 Lossless Data Compression Standard 76 3.2.1 Preprocessor 76 3.2.2 Adaptive entropy encoder 78 3.2.2.1 Variable-length coding 78 3.2.2.2 Coding options 80 3.2.2.3 Coded dataset format 81 3.2.3 Performance evaluation 81 3.2.3.1 1D data: Goddard High-Resolution Spectrometer 82 3.2.3.2 1D data: Acousto-Optical Spectrometer 83 3.2.3.3 1D data: Gamma-Ray Spectrometer 83 3.2.3.4 2D image: Landsat Thematic Mapper 84 3.2.3.5 2D image: Heat-Capacity-Mapping Radiometer 84 3.2.3.6 2D image: Wide-Field Planetary Camera 85 3.2.3.7 2D image: Soft X-Ray Solar Telescope 85 3.2.3.8 3D image: hyperspectral imagery 85 3.3 Image Data Compression Standard 86 3.3.1 Features of the standard 86 3.3.2 IDC compressor 87 3.3.3 Selection of compression options and parameters 91 3.3.3.1 Segment headers 92 3.3.3.2 Integer or float DWT 93 3.3.3.3 Parameters for controlling compression ratio and quality 93 3.3.3.4 Parameters for lossless compression 93 3.3.3.5 Segment size S 94 3.3.3.6 Golomb code parameter 94 3.3.3.7 Custom subband weight 95 3.3.4 Performance evaluation 95 3.3.4.1 Lossless compression results 95 3.3.4.2 Lossy compression results 97 3.4 Lossless Multispectral/Hyperspectral Compression Standard 98 3.4.1 Compressor composition 98 3.4.2 Adaptive linear predictor 99 3.4.3 Encoder 102 3.4.4 Performance evaluation 103 References 104 viii Contents 4 Vector Quantization Data Compression 107 4.1 Concept of Vector Quantization Compression 107 4.2 Review of Conventional Fast Vector Quantization Algorithms 110 4.3 Fast Vector-Quantization Algorithm Based on Improved Distance to MDP 112 4.3.1 Analysis of the generalized Lloyd algorithm for fast training 113 4.3.2 Fast training based on improved distance to MDP 115 4.3.3 Experimental results 117 4.3.4 Assessment of preservation of spectral information 120 4.4 Fast Vector Quantization Based on Searching Nearest Partition Sets 123 4.4.1 Nearest partition sets 124 4.4.2 Upper-triangle matrix of distances 126 4.4.3 p-least sorting 127 4.4.4 Determination of NPS sizes 128 4.4.5 Two fast VQ search algorithms based on NPSs 130 4.4.5.1 Algorithm 1 130 4.4.5.2 Algorithm 2 132 4.4.6 Experimental results 133 4.4.7 Comparison with published fast search methods 136 4.5 3D VQ Compression Using Spectral-Feature-Based Binary Code 138 4.5.1 Spectral-feature-based binary coding 138 4.5.2 Fast 3D VQ using the SFBBC 140 4.5.3 Experimental results of the SFBBC-based VQ compression algorithm 141 4.6 Correlation Vector Quantization 143 4.6.1 Process of CVQ 143 4.6.2 Performance of CVQ 146 4.7 Training a New Codebook for a Dataset to Be Compressed 147 4.8 Multiple-Subcodebook Algorithm Using Spectral Index 149 4.8.1 Spectral indices and scene segmentation 149 4.8.1.1 Manual multithresholding 150 4.8.1.2 Isoclustering 151 4.8.1.3 Histogram-based segmentation with same-size regions 151 4.8.1.4 Modified histogram-based segmentation 152 4.8.2 Methodology of MSCA 153 4.8.3 Improvement in processing time 154 4.8.4 Experimental results of the MSCA 154 4.8.5 MSCA with training set subsampling 157 4.8.6 MSCA with training set subsampling plus SFBBC codebook training 160 4.8.7 MSCA with training set subsampling plus SFBBC for both codebook training and coding 162 Contents ix 4.9 Successive Approximation Multistage Vector Quantization 162 4.9.1 Compression procedure 162 4.9.2 Features 164 4.9.3 Test results 167 4.10 Hierarchical Self-Organizing Cluster Vector Quantization 168 4.10.1 Compression procedure 168 4.10.2 Features 170 References 171 5 Onboard Near-Lossless Data Compression Techniques 177 5.1 Near-Lossless Satellite Data Compression 177 5.2 Cluster SAMVQ 178 5.2.1 Organizing continuous data flow into regional datacubes 178 5.2.2 Solution for overcoming the blocking effect 180 5.2.3 Removing the boundary between adjacent regions 181 5.2.4 Attaining a fully redundant regional datacube for preventing data loss in the downlink channel 182 5.2.5 Compression performance comparison between SAMVQ and cluster SAMVQ 184 5.3 Recursive HSOCVQ 185 5.3.1 Reuse of codevectors of the previous region to attain a seamless conjunction between regions 185 5.3.2 Training codevectors for a current frame and applying them to subsequent frames 186 5.3.3 Two schemes of carrying forward reused codevectors trained in the previous region 188 5.3.4 Compression performance comparison between baseline and recursive HSOCVQ 190 5.4 Evaluation of Near-Lossless Performance of SAMVQ and HSOCVQ 191 5.4.1 Evaluation method and test dataset 191 5.4.2 Evaluation of a single spectrum 192 5.4.3 Evaluation of an entire datacube 194 5.5 Evaluation of SAMVQ with Regard to the Development of International Standards of Spacecraft Data Compression 197 5.5.1 CCSDS test datasets 198 5.5.2 Test results of hyperspectral datasets 199 5.5.3 Compression of multispectral datasets using SAMVQ 203 References 209 6 Optimizing the Performance of Onboard Data Compression 211 6.1 Introduction 211 6.2 The Effect of Raw Data Anomalies on Compression Performance 212 6.2.1 Anomalies in the raw hyperspectral data 212 6.2.2 Effect of spikes on compression performance 213