Data Compression in Spectroscopy Data Compression in Spectroscopy By Joseph Dubrovkin Data Compression in Spectroscopy By Joseph Dubrovkin This book first published 2022 Cambridge Scholars Publishing Lady Stephenson Library, Newcastle upon Tyne, NE6 2PA, UK British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library Copyright © 2022 by Joseph Dubrovkin All rights for this book reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior permission of the copyright owner. ISBN (10): 1-5275-8620-0 ISBN (13): 978-1-5275-8620-8 TABLE OF CONTENTS Preface ........................................................................................................ ix About the Structure of the Book ................................................................ xii Abbreviations ........................................................................................... xiv PART I: Signals and Noise in Spectroscopy Introduction ................................................................................................. 2 Chapter One ................................................................................................. 3 Peak Shapes in Spectroscopy Chapter Two ................................................................................................ 7 Analysis of Noise in Spectral Measurements Chapter Three ............................................................................................ 14 Information-Theoretic Aspects of the Linearly Transformed Spectra Chapter Four .............................................................................................. 24 Big Data Approach to Analytical Spectroscopy PART II: INTRODUCTION TO DATA COMPRESSION Introduction ............................................................................................... 36 Chapter One ............................................................................................... 37 Entropy-Based Compression Chapter Two .............................................................................................. 48 Image Denoising Chapter Three ............................................................................................ 64 Image Compression vi Table of Contents Chapter Four .............................................................................................. 82 Spectral Imaging Chapter Five .............................................................................................. 89 Compressed Sensing PART III: LINEAR TRANSFORMS-BASED ONE DIMENSIONAL ANALYTICAL SPECTROMETRY Introduction ............................................................................................. 100 Chapter One ............................................................................................. 101 General Concepts of the Linear Transforms-Based Analytical Spectrometry Chapter Two ............................................................................................ 111 Interferogram-Based Methods in Spectroscopy Chapter Three .......................................................................................... 119 Fourier Transform-Based Methods in Spectroscopy Chapter Four ............................................................................................ 134 Wavelet Transform-Based Methods in Spectroscopy Chapter Five ............................................................................................ 142 Orthogonal Polynomials in Analytical Spectrometry Chapter Six .............................................................................................. 150 Walsh-Hadamard Transform-Based Methods in Spectroscopy Chapter Seven .......................................................................................... 157 Lineal Transformations of Multivariate Calibration Models Chapter Eight ........................................................................................... 177 Representation of Spectra by Splines PART IV: LINEAR TRANSFORMS-BASED MULTIDIMENSIONAL ANALYTICAL SPECTROMETRY Introduction ............................................................................................. 188 Data Compression in Spectroscopy vii Chapter One ............................................................................................. 190 Correlation Spectroscopy Chapter Two ............................................................................................ 203 Compressed Sensing in Spectroscopy Chapter Three .......................................................................................... 215 Compression of Multidimensional Data Chapter Four ............................................................................................ 222 Matrix Compression Based on the Low Rank Approximation Combined with the FFT of the Singular Vectors Appendix A Signal Processing Using Linear Transforms and Splines A1 ...................................................................................................... 236 Continuous Fourier transform A2 ...................................................................................................... 237 Discrete Fourier Transform A3 ...................................................................................................... 239 Generalized Fourier series and orthogonal polynomials A4 ...................................................................................................... 241 Walsh-Hadamard Transform A5 ...................................................................................................... 242 Splines A6 ...................................................................................................... 244 Two-dimensional Fourier transform A7 ...................................................................................................... 245 Background suppression using DFT A8 ...................................................................................................... 246 Wavelets Appendix B A Bit of Math B1. ...................................................................................................... 250 Singular value decomposition B2. ...................................................................................................... 252 Regularization Appendix C .............................................................................................. 255 Noise viii Table of Contents Appendix D ............................................................................................. 256 Smoothing and Differentiation Digital Filters Appendix E .............................................................................................. 258 Entropy and Information Appendix F .............................................................................................. 260 Spectroscopy Appendix G ............................................................................................. 273 Hyphenated Technique Appendix H ............................................................................................. 275 Matlab Software Bibliography ............................................................................................ 309 Index ........................................................................................................ 353 PREFACE In recent decades, the huge library cabinets filled with books and magazines have been successfully replaced by autonomous and cloud- based electronic digital data storage. High-speed information transmission channels and high-speed computers have made it possible to transfer, accumulate, and process unattainable arrays of numbers, symbols and pictures. Suffice it to mention the Google search engine, which in the middle of the last century could be the subject of fantastic fiction. An inexperienced reader might think that the competition between the amount of information accumulated by mankind and the ability to efficiently archive it is won by the latest technical data storage facilities. Unfortunately, this statement is incorrect; the information ocean is threatened with a new flood but in the intellectual sphere. Where is the exit? To separate the wheat from the chaff [Matthew 13:24-30]? But who can decide what information should be filtered out and what information should be kept? A reasonable solution to this problem is to compress digital data by mathematical processing. Data compression is based on reducing their redundancy, in other words, attenuating information noise (not relevant information) by compromising the fidelity of the useful signal and the level of residual noise. For example, a compressed image requires much less memory to store it than the original, and the deterioration of its quality is not noticeable to the naked eye. For the first time, spectroscopic data, compressed by linear transforms (Fourier, Walsh-Hadamard) has been used to improve the efficiency of quantitative analysis [1] and to simplify the multivariate calibration in NIR spectroscopy [2 and ref. within]. The singular value decomposition of matrix data and use of the most informative principal components is the basic idea of chemometrics [3, 4]. In all these methods, the transformed data was not restored to the standard coordinate system. The main idea of compressed sensing spectral imaging is to perform optical transforms during data acquisition [5]. Now, compression of spectroscopy data is becoming an actual task since this processing allows effective storing and transferring huge datasets of multicomponent mixtures. The reconstructed matrix was successfully used for multivariate calibration and segment cross-validation, clearly demonstrating the potential of the proposed method for future applications x Preface to chemometrics-enhanced spectrometric analysis with limited options of memory size and data transfer rate [6]. Also, it was demonstrated that compression is a perspective method for the economical storage of spectral databases. The present study is an attempt to give a detailed explanation of the reached theoretical and numerical results of data compression in spectroscopy. Since the reconstructed signal contains less noise than the original one, denoising is considered in parallel with compression. While preparing the book, the author faced serious problems associated with its versatility, including the theory and technique of processing one and multidimensional signals, a description of optical devices, and applied physical-chemical tasks of atomic and molecular spectroscopy. The presentation of this material required the application of rather complex mathematical methods, usually little familiar to specialists in analytical spectroscopy. The goal (as in our previous books [7, 8]) was to avoid, where possible, readers' blind faith in the validity of conclusions and recommendations. Theoretical discussions on this issue are illustrated by various examples supplied by a simple program code on MATLAB, which non-professional users can easily modify. The readers who may wish to study the problem further can validate numerical data, given in the book, using computer calculations. Thus, they will be able to understand the details of the algorithm and, if necessary, modify computer programs. References 1. Dubrovkin, J. (1985). Theory and use of the method of linear transformation of spectral coordinates in physicochemical studies. Izvestia Severo-Kavkazskogo Nauchnogo Centra Vysšey Školy, Yestestvennye Nauki, 2, 51-57 [Russian]. 2. Dubrovkin, J. (2017). Linear transformations of multivariate calibration models in near infrared spectroscopy: A Comparative Study. Journal of Near Infrared Spectroscopy, 25, 223-230. 3. Workman Jr., J. J., Mobley, P. R., Kowalski, B. R., Bro, R. (1996). Review of Chemometrics Applied to Spectroscopy: 1985-95, Part 1. Applied Spectroscopy Reviews, 31, 73-124. 4. Mobley, P. R., Kowalski, B. R., Workman Jr., J. J., Bro, R. (1996). Review of Chemometrics Applied to Spectroscopy: 1985-95, Part 2. Applied Spectroscopy Reviews, 31, 347-368. 5. Gamez, G. (2016). Compressed sensing in spectroscopy for chemical analysis. Journal of Analytical Atomic Spectrometry, 31, 2165-2174.