E-AMOM: AN ENERGY-AWARE MODELING AND OPTIMIZATION METHODOLOGY FOR SCIENTIFIC APPLICATIONS ON MULTICORE SYSTEMS A Dissertation by CHARLES WESLEY LIVELY III Submitted to the Office of Graduate Studies of Texas A&M University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY May 2012 Major Subject: Computer Engineering E-AMOM: An Energy-Aware Modeling and Optimization Methodology for Scientific Applications on Multicore Systems Copyright 2012 Charles Wesley Lively III E-AMOM: AN ENERGY-AWARE MODELING AND OPTIMIZATION METHODOLOGY FOR SCIENTIFIC APPLICATIONS ON MULTICORE SYSTEMS A Dissertation by CHARLES WESLEY LIVELY III Submitted to the Office of Graduate Studies of Texas A&M University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Approved by: Chair of Committee, Valerie Elaine Taylor Committee Members, Karen Butler-Purry Eun Jung Kim Tiffani Williams Head of Department, Duncan Walker May 2012 Major Subject: Computer Engineering iii ABSTRACT E-AMOM: An Energy-Aware Modeling and Optimization Methodology for Scientific Applications on Multicore Systems. (May 2012) Charles Wesley Lively III, B.S.E., Mercer University; M.S., Texas A&M University Chair of Advisory Committee: Dr. Valerie Elaine Taylor Power consumption is an important constraint in achieving efficient execution on High Performance Computing Multicore Systems. As the number of cores available on a chip continues to increase, the importance of power consumption will continue to grow. In order to achieve improved performance on multicore systems scientific applications must make use of efficient methods for reducing power consumption and must further be refined to achieve reduced execution time. In this dissertation, we introduce a performance modeling framework, E-AMOM, to enable improved execution of scientific applications on parallel multicore systems with regards to a limited power budget. We develop models for each application based upon performance hardware counters. Our models utilize different performance counters for each application and for each performance component (runtime, system power consumption, CPU power consumption, and memory power consumption) that are selected via our performance-tuned principal component analysis method. Models developed through E-AMOM provide insight into the performance characteristics of iv each application that affect performance for each component on a parallel multicore system. Our models are more than 92% accurate across both Hybrid (MPI/OpenMP) and MPI implementations for six scientific applications. E-AMOM includes an optimization component that utilizes our models to employ run-time Dynamic Voltage and Frequency Scaling (DVFS) and Dynamic Concurrency Throttling to reduce power consumption of the scientific applications. Further, we optimize our applications based upon insights provided by the performance models to reduce runtime of the applications. Our methods and techniques are able to save up to 18% in energy consumption for Hybrid (MPI/OpenMP) and MPI scientific applications and reduce the runtime of the applications up to 11% on parallel multicore systems. v DEDICATION This dissertation is dedicated to my loving and supportive parents, Charles W. Lively Jr. and Irene S. Lively. vi ACKNOWLEDGEMENTS Graduate school has been an enriching journey that has helped to shape my life and thought process. For this, I would first like to give thanks to my Lord and Savior Jesus Christ for guiding me through out this journey and in life. My mother, Irene, has been an endless source of encouragement and support since birth and I could never repay her for always encouraging my intellectual interests. My family and friends have been a great support system over the years and so I would like to thank them for always offering kind words of encouragement and support (Charles Lively Jr., Vidal Lively, Charles Beverley Jr., Courtney Carey, Jesse Dukes, Jacqueline Hodge, Carla Marsh, and Antoinette Davis) I have to thank my academic family for support throughout this time. My advisor and mentor, Dr. Valerie Taylor, has taught me what it truly takes to be an excellent researcher through constant encouragement, hard work, and “refinement”. I would also like to give thanks to my second advisor, Dr. Xingfu Wu, for always providing encouraging feedback and support. Special thanks are also in order for my past research group members Dr. Ayodeji Coker and Dr. Sameh Sharkawi. vii NOMENCLATURE MPI Message Passing Interface HPC High Performance Computing DVFC Dynamic Voltage and Frequency Scaling DCT Dynamic Concurrency Throttling viii TABLE OF CONTENTS Page ABSTRACT .............................................................................................................. iii DEDICATION .......................................................................................................... v ACKNOWLEDGEMENTS ...................................................................................... vi NOMENCLATURE ................................................................................................. vii TABLE OF CONTENTS .......................................................................................... viii LIST OF FIGURES .................................................................................................. x LIST OF TABLES .................................................................................................... xv 1. INTRODUCTION .............................................................................................. 1 1.1 Research Challenges on Multicore Systems ........................................ 8 1.2 Modeling Infrastructure ....................................................................... 11 1.3 Related Work ....................................................................................... 16 2. PROPOSED PERFORMANCE MODELING SCHEME .................................. 27 2.1 Energy-Aware Modeling and Optimization Methodology (E-AMOM) 27 2.2 Performance-Tuned Principal Component Analysis Method .............. 30 2.3 Application Optimization Methods ...................................................... 34 2.4 Modeling Approaches Leveraged ........................................................ 37 3. PERFORMANCE-POWER TRADE-OFFS OF MPI AND HYBRID APPLICATIONS ................................................................................................ 50 3.1 Parallel Multicore Systems .................................................................. 50 3.2 Experimental Environment .................................................................. 52 3.3 Experimental Results ........................................................................... 53 3.4 Summary .............................................................................................. 68 ix Page 4. POWER-AWARE PERFORMANCE MODELS OF SCIENTIFIC APPLICATIONS………………………………………………......…………….. 70 4.1 Performance-Tuned Principal Component Analysis Methodology ..... 70 4.2 HPC Applications ................................................................................ 81 4.3 Experimental Results ........................................................................... 85 4.4 Summary .............................................................................................. 149 5. OPTIMIZATION OF HYBRID AND MPI SCIENTIFIC APPLICATIONS .... 152 5.1 Software-Based Power Reduction Methods ........................................ 152 5.2 Optimization Methodology for Application Kernels ........................... 153 5.3 Loop Optimizations ............................................................................. 160 5.4 Experimental Results ........................................................................... 161 5.5 Summary .............................................................................................. 184 6. SUMMARY AND FUTURE WORK ................................................................ 185 6.1 Summary .............................................................................................. 185 6.2 Future Work ......................................................................................... 186 REFERENCES ......................................................................................................... 189 VITA ......................................................................................................................... 199