Thermal-Aware File and Resource Allocation in Data Centers by Ajit Chavan A dissertation proposal submitted to the Graduate Faculty of Auburn University in partial fulfillment of the requirements for the Degree of Doctor of Philosophy Auburn, Alabama December 16, 2017 Keywords: Thermal-aware, File Allocation, Scheduling, Storage Systems, Hadoop Copyright 2017 by Ajit Chavan Approved by Xiao Qin, Professor of Computer Science and Software Engineering Saad Biaz, Professor of Computer Science and Software Engineering Wei-Shinn Ku, Associate Professor of Computer Science and Software Engineering Sanjeev Baskiyar, Professor of Computer Science and Software Engineering Abstract After addressing the issue of reducing power consumption by computing nodes in data centers, in recent years, computer scientists are focusing on reducing cooling cost of the data centers, thereby making the data centers thermal-aware. Due to the dramatic increase in power-density of data centers, thermal-management strategies are gaining more and more significance in the area of high performance computing data centers. Most of the previous studies towards achieving this goal focus on the nodes performing computation-intensive tasks, in which, the processors are the major consumer of the node’s power. However, there is a lack of study on the ways of thermal-aware data placement in storage clusters housing thousands of storage nodes, where the disk subsystems are the major power consumer. In this dissertation, we propose thermal-aware file and resource allocation policies to reduce the cooling cost of data centers. We first propose a thermal-aware file assignment policy -TIGER- to make file assignments in distributed storage clusters, followed by out re- source allocation policy- TASH, which addresses the issue of thermal management in specific framework (Hadoop). Our proposed thermal-aware policies utilize nodes’ contributions to- wards heat re-circulation in data centers while making file and resource allocation decisions. The proposed policies make use of cross-interference matrix to calculate node’s contribution in heat re-circulation. Our experimental results show that the proposed policies significantly reduce the cooling cost of data centers as compared to existing thermal-aware policies, and at the same time, maintains performance penalties within acceptable margin. ii Acknowledgments Thereisalonglistofpeoplewhocontributedinonewayortheotherduringmygraduate studies at Auburn. First of all, I would like to express my deepest appreciation to Dr. Xiao Qin for being a great advisor, an excellent teacher, and an awesome person. I got to learn a lot of things from him which not only helped me in improving my technical skills but also helped me in becoming a good person. Without his continuous guidance, creative vision, and constant supervision, this research would have never been possible. Every chapter of this dissertation is influenced by him in some way or another. It has been a great honor and pleasure for me to work under Dr. Qin’s supervision. I would also like to thank Dr. Ku, Dr. Baskiyar, and Dr. Biaz for serving on my dis- sertation committee, reviewing the original version of my dissertation, and giving insightful suggestions, which helped me improve many chapters in the dissertation. I am indebted to Dr. Sushil Bhavnani, who served as Faculty advisor for Indian Student Association for all the three years I was in the executive committee. I got to learn many things from him - accepting new culture while not forgetting who we are, importance of punctuality, and planning big events - are few of them. IwouldliketoexpressmygratitudetoDr. JiZhangandDr. XunfeiJiangforhelpingme understand many technical aspects such as cluster setup, lab network topology and many more things at the beginning of my graduate studies. I would also like to thank Sanjay Kulkarni, for helping me understand the source code of Hadoop framework. Sanjay is also a very good friend and a peer reviewer. I am also thankful to all my colleagues at Computer Systems Lab, in particular, Tausif Muzaffar, Yuanqi Chen, Chetan Sonami for their help and suggestions. iii I am deeply indebted to Vikalp Narayan and Aditya Singh - best friends and awesome roommates. They were always there to help me with all of my big decisions and have supported me in my emotional low points. I am grateful to Jasma didi, Bhumi ben, Avanti Kulkarni, Niranjana Aunty for ensuring that I missed none of the home food, and Sadhwi Ravichandran for offering me great suggestions in my social life and for South Indian culture and food. I would also like to thank Vishal Kothari, Kunal Sevak, and Gayatri didi for playing a role of elder siblings. I must thank Vibudh Mishra, Adarsh Jain, Digvijay Gholap, and Mikhail Zade for helping me in the vital beginning of my life as a graduate student. In addition to this, I owe a debt of gratitude to my friends Nakul Kothari, Shantanu Desh- pande, Vijith Beemireddi, Robin Muthukumar, Amey Rane, Prachi Sangle, Aditya Agrawal, Shubham Garg, Micah Bowden for all their support, laughs and great memories. I would also like to thank all ISA committee members. Most of all, would like to express my deepest gratitude to my parents. You have sac- rificed a lot so that I can pursue my dreams and words cannot express my feelings nor my gratitude for all your love and compassion. I am who I am, only because of you. My thanks also goes to my elder brother Abhay Chavan for being my brother, friend, teacher, advisor, and role model in my personal life. I must thank Nivas kaka, Sushma aatya, Mai aatya, Amey dada, Sachin dada, and Chaitrali tai for my unforgettable childhood memories. iv Table of Contents Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1.1 New Trends . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1.2 Thermal Emergencies in Data Center . . . . . . . . . . . . . . . . . . 4 1.1.3 Limitations of Existing Approaches . . . . . . . . . . . . . . . . . . . 7 1.2 Scope of the Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.4 Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.1 Energy Efficient Data Center Nodes . . . . . . . . . . . . . . . . . . . . . . . 11 2.1.1 Computational Node Energy Savings . . . . . . . . . . . . . . . . . . 12 2.1.2 Energy Savings in Storage Nodes . . . . . . . . . . . . . . . . . . . . 14 2.2 Thermal Efficiency of Data Centers . . . . . . . . . . . . . . . . . . . . . . . 15 2.2.1 Thermal Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.2.2 Thermal Management . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.3 Energy Conservation in MapReduce Clusters . . . . . . . . . . . . . . . . . . 17 2.3.1 Energy Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.3.2 Thermal Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3 Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 v 3.1 Heat Re-circulation Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.2 Cooling Cost Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.3 Power Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 4 Thermal-Aware File Assignment in Storage Clusters . . . . . . . . . . . . . . . . 26 4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.1.1 Creation of New Files . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.1.2 File Migration and Access Pattern Monitor . . . . . . . . . . . . . . . 28 4.2 Design Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 4.3 Disk Utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 4.3.1 Computing Disk Utilization Threshold . . . . . . . . . . . . . . . . . 30 4.3.2 Considering Heat Re-circulation . . . . . . . . . . . . . . . . . . . . . 31 4.4 The File Assignment Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 33 5 Thermal-Aware Scheduling in Hadoop Cluster . . . . . . . . . . . . . . . . . . . 36 5.1 Revised Power Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 5.2 Apache Hadoop: A MapRedue Framework . . . . . . . . . . . . . . . . . . . 38 5.2.1 The MapReduce Framework . . . . . . . . . . . . . . . . . . . . . . . 38 5.2.2 Hadoop Scheduling Policies . . . . . . . . . . . . . . . . . . . . . . . 39 5.2.3 Impact of Resource Allocation on Power Consumption . . . . . . . . 40 5.3 TASH : Thermal-Aware Scheduling in Hadoop . . . . . . . . . . . . . . . . . 41 5.3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.3.2 The Thermal-Aware Scheduler . . . . . . . . . . . . . . . . . . . . . . 43 5.4 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 5.4.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 5.4.2 Thermal-Aware Yarn . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 6 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 6.1 TIGER Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 6.1.1 Baseline Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 vi 6.1.2 Experimental Setup and Workload Characteristics . . . . . . . . . . . 53 6.1.3 Thermal Impact of Energy Efficient Disks . . . . . . . . . . . . . . . 54 6.1.4 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 6.1.5 Impact of Initial Supply Temperature . . . . . . . . . . . . . . . . . . 61 6.2 TASH Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 6.2.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 6.2.2 TASH performance on MapReduce Benchmarks . . . . . . . . . . . . 66 6.2.3 Impact of Resource Allocation on Power Consumption . . . . . . . . 69 7 Extension of File and Resource Allocation Policies . . . . . . . . . . . . . . . . . 73 7.1 Extension of File Assignment Policy . . . . . . . . . . . . . . . . . . . . . . . 73 7.1.1 Anomalous Behavior of TIGER . . . . . . . . . . . . . . . . . . . . . 73 7.1.2 HybridTIGER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 7.1.3 Evaluation of HybridTIGER . . . . . . . . . . . . . . . . . . . . . . . 76 7.2 TASH in MR1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 7.2.1 MR1 Internals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 7.2.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 7.2.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 8 Conclusion and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 8.1 Main Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 8.1.1 Theraml-Aware File Assignment . . . . . . . . . . . . . . . . . . . . . 84 8.1.2 Thermal-aware Resource Allocation . . . . . . . . . . . . . . . . . . . 85 8.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 8.2.1 Data Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 8.2.2 Machine learning in thermal management . . . . . . . . . . . . . . . 87 8.2.3 Thermal Management in Other Applications . . . . . . . . . . . . . . 87 8.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 vii List of Figures 1.1 Data center layout with hot and cold aisles and CRAC. . . . . . . . . . . . . . . 4 2.1 A simplified taxonomy of the approaches to the energy conservation problem in data centers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 4.1 TIGER:System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 5.1 The system architecture of TASH, which integrates (1) a schedule, (2) the cross- interference matrix (CIM), and (3) the main module (a.k.a., resource manager). 42 6.1 Data Center Layout [1] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 6.2 Inlet temperatures and cooling cost of CoolestInlet, SortPart, and TIGER al- gorithms in scenario 1, where idle disks are transitioned into the sleep mode to conserve energy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 6.3 Inlet temperatures and cooling cost of CoolestInlet, SortPart, and TIGER algo- rithms in scenario 2, where idle disks are never transitioned into the sleep mode. 57 6.4 ImpactofsleepmodepercentageoninlettemperaturesandcoolingcostofCoolestIn- let, SortPart, and TIGER. The system utilization is kept at 50% . . . . . . . . 58 6.5 Impact of the number of nodes on maximum inlet temperature and cooling cost. 60 6.6 Total power consumption (i.e., cooling and operational cost) of the data cen- ter managed by TIGER, CoolestInlet, and SortPart in a) Scenario 1 (see Sec- tion 6.1.3), b) Scenario 2 (see Section 6.1.3), and c)sleep mode percentage 50% . 61 6.7 Impact of initial supply temperature on maximum inlet temperature and cooling cost. Initial T = 120C . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 sup 6.8 Impact of initial supply temperature on maximum inlet temperature and cooling cost. Initial T = 13.50C . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 sup 6.9 Average response time of the data center managed by TIGER, CoolestInlet, and SortPart. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 6.10 Cross Interference Coefficient of Data Center . . . . . . . . . . . . . . . . . . . 66 6.11 Inlet temperatures and cooling cost of YARN and TASH in for different bench- marks. Input size 100GB for each benchmark. . . . . . . . . . . . . . . . . . . . 67 viii 6.12 Average runtime for benchmarks. . . . . . . . . . . . . . . . . . . . . . . . . . . 68 6.13 Cross Interference Coefficient of Data Center . . . . . . . . . . . . . . . . . . . 69 6.14 Figure depicting number of containers assigned and power consumed by each node in a rach under YARN and TASH. . . . . . . . . . . . . . . . . . . . . . . 70 6.15 CPU utilization of Original Yarn and TASH for running TeraSort. . . . . . . . . 71 7.1 The number of disks with minimum utilization and the number of disks with maximum utilization. (a) Minimum utilization range: from U to (U +0.1) min min (b) Maximum utilization range: from U -0.1 to U . . . . . . . . . . . . . . 74 max max 7.2 Inlet temperatures and cooling cost of CoolestInlet, TIGER, and HybridTIGER in scenario 1, where idle disks are transitioned into the sleep mode to offer energy savings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 7.3 tHadoop: System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 ix List of Tables 2.1 Comparison of TIGER and the existing solutions . . . . . . . . . . . . . . . . . 17 3.1 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 5.1 A list of configuration parameters for YARN. . . . . . . . . . . . . . . . . . . . 48 6.1 Cluster Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 6.2 Hadoop Benchmarks Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 7.1 Properties in core-site.xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 x