Energy Adaptive Digital Ecosystems by Andreas Christoph Bergen M.Sc., Computer Science, University of Victoria, 2013 B.Sc., Computer Science, University of Victoria, 2011 B.A., Political Science and History, University of Victoria, 2007 A Dissertation Submitted in Partial Fulfillment of the Requirements for the Degree of DOCTOR OF PHILOSOPHY in the Department of Computer Science (cid:13)c Andreas Christoph Bergen, 2017 University of Victoria All rights reserved. This dissertation may not be reproduced in whole or in part, by photocopying or other means, without the permission of the author. ii Energy Adaptive Digital Ecosystems by Andreas Christoph Bergen M.Sc., Computer Science, University of Victoria, 2013 B.Sc., Computer Science, University of Victoria, 2011 B.A., Political Science and History, University of Victoria, 2007 Supervisory Committee Dr. H. A. Mu¨ller, Supervisor (Department of Computer Science) Dr. Y. Coady, Supervisor (Department of Computer Science) Dr. S. Neville, Outside Member (Department of Electrical and Computer Engineering) iii ABSTRACT Since the turn of the century, the proliferation of virtualization and cloud computing has led to an increase in data centres and consequently an increase in power con- sumption for computing. Today, approximately 2% of global energy consumption is attributed to data centres alone. As a result, optimizing power usage effectiveness in enterprise data centres has become a laudable goal and a critical requirement in IT operations all over the world. While a significant body of research exists to mea- sure, monitor, and control the “greenness” level of hardware components, significant research is needed to relate hardware energy consumption to energy consumption stemming from (software) program execution. In this dissertation, we argue that the true energy cost of program execution must focus on the digital ecosystem within whichaparticularsoftwareprogramisexecuted. Weinvestigatetheinterplaybetween energy consumption, task scheduling and execution decision making using dynamic runtime models of digital ecosystems based on the execution context of software. Single instances of software applications are no longer confined to a single de- vice or machine. Instead software commonly interacts with resources and services outside of its own hardware unit. The scope of this interaction defines the applica- tion’s digital ecosystem. Smartphones interact with cloud resources; cloud resources include databases, specialized compute or storage clouds, specialized hardware and virtual machines (VMs). Combining processes of varying complexity with varying resource allocations produces different energy consumption levels. The challenge is to investigate the variability of software process orchestration based on a power con- sumption framework to accrue and optimize energy savings in digital ecosystems. The contributions of this dissertation include: i) an adaptive energy consumption framework; ii) self-adaptive energy management systems based on this framework; iii) deployment mechanisms for applications to use this framework; iv) models at runtime models for self-adaptive energy management systems. Our ultimate goal is to develop smart, self-adaptive, green computing techniques, such as adaptive job scheduling and resource provisioning, to reduce overall power consumption in data centres, on individual devices (e.g., mobile, desktop, laptop or server), and in digital ecosystems. iv Contents Supervisory Committee ii Abstract iii Table of Contents iv List of Figures viii Acknowledgements xi Dedication xii 1 Introduction and Motivation 1 1.1 Motivating Software Energy Consumption and its Optimization . . . 2 1.2 Our Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2.1 Overview of our Approach . . . . . . . . . . . . . . . . . . . . 6 1.2.2 Evolution to Digital Ecosystems . . . . . . . . . . . . . . . . . 8 1.2.3 Dimensions of Energy Consumption Research . . . . . . . . . 10 1.3 Evolution of Energy Optimization . . . . . . . . . . . . . . . . . . . . 10 1.4 Research Questions and Contributions . . . . . . . . . . . . . . . . . 13 1.4.1 Exploratory Questions . . . . . . . . . . . . . . . . . . . . . . 13 1.4.2 Design Questions . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.5 Dissertation Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2 Research Statement and Approach 16 2.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.2 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.2.1 Exploratory Questions . . . . . . . . . . . . . . . . . . . . . . 19 2.2.2 Design Questions . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.3 Long Term Research Goal . . . . . . . . . . . . . . . . . . . . . . . . 23 v 3 Research Background 24 3.1 Green Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.1.1 VM Consolidation and Heuristics . . . . . . . . . . . . . . . . 25 3.1.2 Networking Components . . . . . . . . . . . . . . . . . . . . . 27 3.1.3 Software Metering and Instrumentation . . . . . . . . . . . . . 28 3.1.4 Instruction Level Analysis . . . . . . . . . . . . . . . . . . . . 29 3.1.5 Higher Level Instruction Analysis . . . . . . . . . . . . . . . . 29 3.1.6 Energy Consumption and Markets . . . . . . . . . . . . . . . 30 3.1.7 Forecasting Energy Consumption . . . . . . . . . . . . . . . . 31 3.1.8 Simulation versus Execution Environment . . . . . . . . . . . 31 3.2 Software Engineering for Self-Adaptive and Self-Managing Systems . 32 3.2.1 Self-Adaptive and Self-Managing Systems . . . . . . . . . . . 32 3.2.2 Data Science—Machine Learning and Statistical Analysis . . . 36 3.3 Control Engineering and Feedback Loops . . . . . . . . . . . . . . . . 38 3.4 Simulation, Signal Processing and Statistical Analysis . . . . . . . . . 43 3.4.1 Simulating Energy Consumption in Data Centres . . . . . . . 46 3.4.2 Signal Processing and Statistical Analysis . . . . . . . . . . . 47 3.5 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 4 Research Methodology 50 4.1 Process and Hypothesis Formulation . . . . . . . . . . . . . . . . . . 51 4.1.1 Initial Hypothesis . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.1.2 Hypothesis Refinement and Iterative Approach . . . . . . . . 52 4.2 Data—Sources, Collection and Analysis . . . . . . . . . . . . . . . . . 52 4.2.1 Data Sources . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.2.2 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.2.3 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 4.3 Dynamic Models at Runtime . . . . . . . . . . . . . . . . . . . . . . . 56 4.3.1 Model Design Process . . . . . . . . . . . . . . . . . . . . . . 56 4.4 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 4.5 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 5 Experiments 60 5.1 Our Approach to Energy Consumption Experiments . . . . . . . . . . 60 5.2 Balanced Energy Usage and Performance Optimization . . . . . . . . 63 vi 5.3 Experiments—Description and Process . . . . . . . . . . . . . . . . . 63 5.3.1 Benchmarking and Automatic Profiling . . . . . . . . . . . . . 64 5.3.2 Disk Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . 66 5.3.3 Memory Benchmarks . . . . . . . . . . . . . . . . . . . . . . . 73 5.3.4 CPU Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . 76 5.3.5 Network Benchmarks with Limitations . . . . . . . . . . . . . 80 5.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 5.5 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 6 Effects on Energy Consumption 83 6.1 Energy versus Performance—An Overview . . . . . . . . . . . . . . . 83 6.2 Scheduling Strategies for Energy Consumption . . . . . . . . . . . . . 85 6.2.1 Dynamic Strategies and Scheduling Policies . . . . . . . . . . 89 6.3 Machine Learning and Energy Consumption . . . . . . . . . . . . . . 95 6.3.1 Conducting Experiments . . . . . . . . . . . . . . . . . . . . . 98 6.3.2 Experimental Results and Discussion . . . . . . . . . . . . . . 100 6.3.3 Adaptive Energy Consumption . . . . . . . . . . . . . . . . . 102 6.3.4 Motivation for Adaptive Solutions . . . . . . . . . . . . . . . . 102 6.3.5 Realizing Energy Adaptation in Data Centers . . . . . . . . . 102 6.3.6 MIAC Deployment Architecture . . . . . . . . . . . . . . . . . 103 6.4 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 7 Valuation and Discussion of Results 106 7.1 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 7.1.1 PDU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 7.1.2 Initial Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 7.1.3 Focussing on Finer Level Control . . . . . . . . . . . . . . . . 108 7.2 Avenues for Future Research . . . . . . . . . . . . . . . . . . . . . . . 110 7.2.1 Risks and Progress in PhD Studies . . . . . . . . . . . . . . . 110 7.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 8 Conclusions and Future Research 113 8.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 8.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 8.3 Future Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 vii A Benchmark Sample Code 120 A.1 Sample Code to Copy File with fputs/fgets . . . . . . . . . . . . . . . 120 A.2 Sample Code to Copy File with fputs/fgets . . . . . . . . . . . . . . . 122 A.3 Sample Code Used to Copy File with read/write Large Buffer . . . . 123 A.4 Sample Code Used to Copy File with read/write Small Buffer . . . . 125 A.5 Script to Control CPU and Memory Benchmarks . . . . . . . . . . . 126 A.6 Script to Control CPU . . . . . . . . . . . . . . . . . . . . . . . . . . 130 A.7 Primitive Code for Memory Testing . . . . . . . . . . . . . . . . . . . 131 A.8 Primitive Code for Memory Testing . . . . . . . . . . . . . . . . . . . 132 A.9 Second Script for Memory Testing . . . . . . . . . . . . . . . . . . . . 133 Bibliography 134 viii List of Figures Figure 1.1 Fields and topics of energy consumption research—Courtesy of Lago et al. [77] . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Figure 3.1 Autonomic element with managed system and autonomic man- ager manager featuring the MAPE-K loop with its standard components [63,71] . . . . . . . . . . . . . . . . . . . . . . . . 33 Figure 3.2 Autonomic computing reference architecture (ACRA) [63] . . 35 Figure 3.3 Classic control loop [57] . . . . . . . . . . . . . . . . . . . . . 38 Figure 3.4 Referencearchitectureformodelreferenceadaptivecontrol(MRAC) 42 Figure 3.5 Reference architecture for model identification adaptive control (MIAC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Figure 3.6 DYNAMICO reference model [129] . . . . . . . . . . . . . . . 44 Figure 3.7 Artificially generated, noisy signal to illustrate the effects of Fourier Transform [137]. . . . . . . . . . . . . . . . . . . . . . 45 Figure 3.8 The result of the Fourier Transform applied to Figure 3.7 ex- hibiting distinct frequencies present in a seemingly noisy sig- nal [137]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Figure 3.9 Classification process stages . . . . . . . . . . . . . . . . . . . 48 Figure 4.1 Our MAPE-K model to manage energy consumption and effect energy optimization in data center software . . . . . . . . . . 57 Figure 5.1 Optimizing energy use and performance in digital ecosystems . 61 Figure 5.2 Deployment and execution cycle of automated benchmarking tool 64 Figure 5.3 Simplified hierarchy of scripts used for benchmarking on Linux machines. Each leaf node (CPU, Memory, Disk, wget) can be composed of multiple individual benchmarks. . . . . . . . . . 66 Figure 5.4 Sample code used to copy file with fgetc/fputc . . . . . . . . . 67 Figure 5.5 Sample code used to copy file with fputs/fgets . . . . . . . . . 67 ix Figure 5.6 Sample code used to copy file with read/write large buffer . . 68 Figure 5.7 Sample code used to copy file with read/write small buffer . . 68 Figure 5.8 Energy consumption for different implementations of disk I/O for a fixed amount of time. No optimization flags were used at compile time to produce the executables. Each function was executed for a fixed amount of time, capturing data points as the benchmark completes. . . . . . . . . . . . . . . . . . . . . 70 Figure 5.9 Complete copies per experiment and costs per write relative to fgets/fputs. Read/Write achieve more completed copies and are cheaper per write. . . . . . . . . . . . . . . . . . . . . . . 71 Figure 5.10 Power profile of a process dominated by memory resource uti- lization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Figure 5.11 Power profile of a process dominated by large memory resource utilization. Memory utilization accounts for more energy con- sumption near the beginning of the benchmark when data is frequently loaded and changed in memory. . . . . . . . . . . . 75 Figure 5.12 Energy consumption over 300 sample points for varying CPU utilization between 2 and 20 cores. Energy consumption in- creases linearly until half the cores are fully utilized. Each set of cores is utilized consistenly for a fixed amount of time. . . . 77 Figure 5.13 Power profile of CPU-intensive benchmarks running at eight cores with eight threads, eight cores with 16 threads, and 16 cores with 16 threads for the same workload. . . . . . . . . . . 78 Figure 6.1 Performance of CPU-intensive reference job on SAVI versus GENIservers. Alessperformantserverintermsofjobduration (SAVI), which consistently consumes less energy for the same job. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 Figure 6.2 Detailed statistics gathered running experiments to compare the performance of the SAVI and GENI servers . . . . . . . . 86 Figure 6.3 Idealized illustration of the relation between energy consump- tionandperformance—whileabstracted,theillustrationisbased on our actual findings . . . . . . . . . . . . . . . . . . . . . . 87 x Figure 6.4 Idealised illustration of relation between energy consumption and performance overlayed with actual measurements in data center using server GeniRack02. We overlay actual measured timeseries results with the idealized curve. . . . . . . . . . . . 88 Figure 6.5 GeniRack01, GeniRack02 and GeniRack04 CPU performance relative to energy consumption. We overlay actual measured timeseries results with the idealized curve. . . . . . . . . . . . 90 Figure 6.6 Comparison of CPU levels of multiple servers relative to mul- tiple servers. The south-east corner of the plot is the ideal job placement as it maximizes performance and minimizes energy consumption. . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 Figure 6.7 Comparison memory and disk bandwidth levels of multiple servers relative to their energy consumption. The south-east corner of the plot is the ideal job placement as it maximizes performance and minimizes energy consumption. . . . . . . . 92 Figure 6.8 ComparisonofCPU,memoryanddiskbandwidthlevelsofmul- tiple servers. The south-east corner of the plot is the ideal job placement as it maximizes performance and minimizes energy consumption. . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 Figure 6.9 Classification System . . . . . . . . . . . . . . . . . . . . . . . 97 Figure 6.10 Experimental data over a period of four days. We run bench- marks to test 2 to 20 CPUs at full utilization for a fixed amount of time as well as our memory usage benchmark. The experi- ment is run repeatedly to confirm consistency and to confirm assumptions regarding measured energy consumption (e.g., the other servers on the rack are not contributing to energy con- sumption changes). . . . . . . . . . . . . . . . . . . . . . . . . 99 Figure 6.11 Classification accuracy rates for all classes corresponding to the number of features with different thresholds. . . . . . . . . . . 101