Distribution Statement A: Approved for Public Release Visible Spectrum Based Non-Contact Detection and Characterization of Blood Wave Signal Dynamics and Applications in Stress Detection A Dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy at George Mason University By Balvinder Kaur Master of Science George Mason University, 2012 Bachelor of Science George Mason University, 1999 Dissertation Director: Vasiliki N. Ikonomidou, Assistant Professor Department of Electrical and Computer Engineering Spring Semester 2016 George Mason University Fairfax, VA Copyright © 2016 by Balvinder Kaur All Rights Reserved ii DEDICATION I dedicate this thesis work to my beloved and supportive husband Harpreet Singh and our intellectual and affectionate son Taaren Veer Singh who encouraged and supported me throughout this journey of the doctorate degree program. Thanks are due to my father-in- law Mr. Jagjit Singh, mother-in-law, Mrs. Narinder Kaur, my father, Mr. Balbir Singh, and my mother, Mrs. Surjit Kaur, whose support and interest made the work much easier. iii ACKNOWLEDGEMENTS Dr. Vasiliki N. Ikonomidou, my advisor, provided me invaluable help, guidance, and support throughout my doctorate research. Her knowledge, flexibility, patience, kindness, mentoring, and friendship were vital to my success. I also gratefully thank and acknowledge my committee members, Dr. Andre Manitius, Dr. Babajide O. Familoni, Dr. Dr. Gerald Cook, and Jill K. Nelson for their help, support, and guidance. Very special thanks are owed to my chain of command for encouraging and supporting higher education starting with Dr. Donald A. Reago, Richard A. Nabors, Mr. William S. Jarvis, Dr. Dr. Keith A. Krapels, Ms. Susan M. Harkrider, Dr. Joseph J Durek, Mr. Jonathan G. Hixson, my colleges and friends for guiding and supporting me throughout my Doctorate program. Finally, I want to thank Dr. James A. Hutchinson for being my mentor, associate, colleague, friend, and an individual who inspired me to find a visible spectrum based solution for remote stress detection. iv TABLE OF CONTENTS Page LIST OF ACRONYMS ____________________________________________________ viii LIST OF TABLES _________________________________________________________ xi LIST OF FIGURES ________________________________________________________ xii ABSTRACT ____________________________________________________________ xvi CHAPTER ONE: INTRODUCTION ____________________________________________ 1 Problem Statement __________________________________________________________ 2 Proposed Solution ___________________________________________________________ 2 Significance ________________________________________________________________ 2 Objectives _________________________________________________________________ 3 Innovation _________________________________________________________________ 4 Organization of Dissertation __________________________________________________ 5 CHAPTER TWO: BACKGROUND AND MOTIVATION ____________________________ 7 Heart Rate Variability (HRV) Analysis based on Contact Sensing ______________________ 8 Obtaining More Accurate RR-Intervals by Noise Removal __________________________________ 8 Heart Rate Variability Analysis for Stress Detection ______________________________________ 10 Heartbeat Signal Detection based on Non-Contact Sensing _________________________ 12 Pulse Transit Time (PTT) and Its Relationship to Blood Pressure _____________________ 16 Physiological Stress Detection based on Non-Contact Sensing ______________________ 19 Motivation ________________________________________________________________ 21 CHAPTER THREE: CHARACTERIZE REMOTELY DETECTED BLOOD WAVE (BW) SIGNAL AND GENERATE RR-INTERVAL SIGNAL _____________________________________ 24 Introduction ______________________________________________________________ 24 Materials and Methods _____________________________________________________ 25 Experimental Setup________________________________________________________________ 26 Experiments _____________________________________________________________________ 27 Remote Blood Wave (BW) Signal Detection ____________________________________________ 29 Factors Impacting a Remote BW Signal ________________________________________________ 36 RR-Interval Estimation from BW Signals _______________________________________________ 38 v Results ___________________________________________________________________ 51 Discussion ________________________________________________________________ 55 CHAPTER FOUR: REMOTE MEASUREMENTS OF HEART RATE VARIABILITY AND ITS USE FOR STRESS DETECTION _________________________________________________ 57 Introduction ______________________________________________________________ 57 Materials and Methods _____________________________________________________ 59 Experiments _____________________________________________________________________ 60 Prepare Data for HRV Analysis _______________________________________________________ 62 Determine HRV-Driven Features for Stress Detection ____________________________________ 63 Classification for Stress Detection ____________________________________________________ 64 Confidence Metrics to Quantify Algorithm Performance __________________________________ 68 Results ___________________________________________________________________ 69 Stress Detection Results using ECG Data _______________________________________________ 71 Stress Detection Results using VIS Data ________________________________________________ 74 Repeatable Stress Detection using VIS Data ____________________________________________ 77 Improvements in Stress Detection Results _____________________________________________ 79 Discussion ________________________________________________________________ 81 CHAPTER FIVE: REMOTE MEASUREMENTS OF DIFFERENTIAL PULSE TRANSIT TIME AND ITS USE FOR STRESS DETECTION __________________________________________ 84 Introduction ______________________________________________________________ 84 Materials and Methods _____________________________________________________ 85 Experiments _____________________________________________________________________ 85 Remote Blood Wave (BW) Signal Detection ____________________________________________ 88 Differential Pulse Transit Time (dPTT) Computations _____________________________________ 88 Remote Stress Detection using dPTT __________________________________________________ 90 Results ___________________________________________________________________ 91 Discussion ________________________________________________________________ 96 CHAPTER SIX: ROBUSTNESS OF REMOTE STRESS DETECTION FROM VISIBLE SPECTRUM VIDEOS ______________________________________________________________ 98 Introduction ______________________________________________________________ 98 Methods and materials _____________________________________________________ 100 Remote Blood Wave (BW) Detection _________________________________________________ 100 Remote Differential Pulse Transit Time (dPTT) Estimation ________________________________ 101 Remote Heart rate variability (HRV) Feature Generation _________________________________ 101 Simulated Data __________________________________________________________________ 102 Robustness of the BW Signal _______________________________________________________ 103 Robustness of the dPTT Metric _____________________________________________________ 104 Robustness of the HRV-driven Features ______________________________________________ 104 Results __________________________________________________________________ 106 vi Discussion and Conclusions _________________________________________________ 112 CHAPTER SEVEN: CONCLUSIONS AND FINAL COMMENTS _____________________ 115 Conclusions ______________________________________________________________ 115 Impact of Dissertation _____________________________________________________ 119 Future Research Areas _____________________________________________________ 120 Final Remarks ____________________________________________________________ 121 REFERENCES _________________________________________________________ 122 vii LIST OF ACRONYMS AAMI Advancement of Medical Instrumentation ANS Autonomic Nervous System ASRT Affective Stress Response Test AUC Area Under the Curve BDC Blind Deconvolution BPL Back Palm Left BPM Back Palm Middle BPR Back Palm Right BW Blood Wave CF Curve Fitting CH Chin CL Cheek Left CR Cheek Right DC Direct Current dPTT Differential Pulse Transit Time DSLR Digital Single-Lens Reflex ECG Electrocardiograph or Electrocardiogram FFT Fast Fourier Transform FL Forehead Left FM Forehead Middle FN False Negative FP False Positive FPL Front Palm Left viii FPM Front Palm Middle FPR Front Palm Right FR Forehead Right FSVM Fuzzy Support Vector Machine GT Ground Truth GUI Graphical User Interphase HB Heartbeat HF High Frequency HPF High Pass Filter HRV Heart Rate Variability HRVAS Heart Rate Variability Analysis Software HRVTi Heart Rate Variability Triangular Index ICA Independent Component Analysis IR Infrared IRB Internal Review Board LDA Linear Discriminant Analysis LDV Laser Doppler Vibrometer LF Low Frequency LFHF Low Frequency over High Frequency LMS Least Mean Squares LNR Light to Noise Ratio LPF Low Pass Filter LR Logistic Regression LS Least Squares LWIR Long-Wave Infrared MSE Mean Square Error MST Modified Stroop Test MWIR Mid-Wave Infrared nHF Normalized to Total Power High Frequency PCA Principal Component Analysis ix