5 . 6 e r e h p S v e r a w M V VMware vSphere 6.5 Host Resources Deep Dive Copyright © 2017 by Frank Denneman and Niels Hagoort All rights reserved. No part of this book shall be reproduced, stored in a retrieval system, or transmitted by any means, electronic, mechanical, or otherwise, without written permission from the publisher. No patent liability is assumed on the use of the information contained herein. Although every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions. Neither is any liability assumed for damages resulting from the use of the information contained herein. International Standard Book Number: ISBN 9781540873064 All terms mentioned in this book that are known to be trademarks or service marks have been appropriately capitalized. Use of a term in this book should not be regarded as affecting the validity of any trademark or service mark. Cover design by DigitalMaterial.nl Version: 1.01 ABOUT THE AUTHORS Frank Denneman is a Senior Staff Architect at VMware R&D focusing on VMware Cloud on AWS. Frank is a VCDX (29) and co-author of the bestselling vSphere Clustering Deep Dive series. Frank presents on a regular basis at global virtualization events and has been a VMworld Top 10 speaker for five consecutive years. You can find his articles at www.frankdenneman.nl. Follow Frank on Twitter @frankdenneman. Niels Hagoort is a freelance Virtualization Architect with more than 15 years of experience. Niels has extensive knowledge in the field of designing, building and implementing complex enterprise IT infrastructures. Niels presents on a regular basis at global virtualization events and is a VCDX (212). You can find his articles at www.nielshagoort.com. Follow Niels on Twitter @NHagoort. ACKNOWLEDGEMENTS .................................................................................. 9 FOREWORD ....................................................................................................... 11 PREFACE ........................................................................................................... 17 CPU RESOURCES ............................................................................................ 19 PROLOGUE ....................................................................................................... 21 CPU ARCHITECTURE ..................................................................................... 23 Insights into ESXi Host Configurations .................................................................... 23 Non-Uniform Memory Access ..................................................................................... 26 From UMA to NUMA ....................................................................................................... 27 Caching Snoop Protocols ............................................................................................... 28 Uniform Memory Access Architecture ..................................................................... 29 Non-Uniform Memory Access Architecture ........................................................... 30 Sufficiently Uniform Memory Architecture ............................................................ 35 CPU Microarchitecture Overview ............................................................................... 38 SYSTEM ARCHITECTURE ............................................................................. 41 Terminology ....................................................................................................................... 41 NUMA Architecture ......................................................................................................... 42 Consuming NUMA ........................................................................................................... 43 BIOS Setting: Node Interleaving .................................................................................. 44 BIOS Setting: ACPI SLIT Preferences ......................................................................... 45 CPU System Architecture .............................................................................................. 46 Uncore .................................................................................................................................. 46 Quick Path Interconnect Link ...................................................................................... 50 QPI Link Speed Impact on Performance .................................................................. 53 Intel QPI Link Power Management ............................................................................ 54 CACHE COHERENCY ...................................................................................... 57 Caching Architecture ...................................................................................................... 58 Data Prefetching ............................................................................................................... 59 Last Level Cache ............................................................................................................... 60 Scalable On-Die Ring ...................................................................................................... 61 Cache Snooping ................................................................................................................ 62 Snoop Modes...................................................................................................................... 64 Cluster-on-Die Architecture ......................................................................................... 67 Snoop Mode Recommendation ....................................................................................71 VMKERNEL NUMA CONSTRUCTS.............................................................. 73 ESXi VMkernel NUMA Constructs ............................................................................. 75 Adjusting Virtual NUMA Topology ............................................................................ 78 Virtual Proximity Domains and Physical Proximity Domains ......................... 86 Guest OS NUMA Optimization ..................................................................................... 95 NUMA Support for CPU Hot Add ................................................................................. 97 NUMA Support for Memory Hot Plug ...................................................................... 103 Advanced Setting numa.consolidate ...................................................................... 103 Size Your VM Correctly ................................................................................................ 105 NUMA SCHEDULER ..................................................................................... 107 NUMA Scheduling Goals ............................................................................................. 107 NUMA Scheduler Initial Placement ........................................................................ 108 NUMA Scheduler Load Balancing ............................................................................ 122 CPU SCHEDULER .......................................................................................... 135 CPU States ........................................................................................................................ 136 CPU Scheduler Accounting .........................................................................................137 Performance Isolation and Fairness ....................................................................... 143 World Placement ............................................................................................................ 148 Load Balancing ............................................................................................................... 149 Intel Hyper-Threading................................................................................................... 151 Co-Scheduling ..................................................................................................................156 Scheduling Affinity ....................................................................................................... 160 Latency Sensitivity Setting ........................................................................................ 164 HOST POWER MANAGEMENT................................................................... 169 Power Management Control ....................................................................................... 169 CPU Power Management Technology ..................................................................... 170 P-States ............................................................................................................................. 172 C-States ............................................................................................................................. 174 Intel Turbo Boost Technology .................................................................................... 180 Turbo Boost Basics ......................................................................................................... 181 Turbo Bins ........................................................................................................................ 182 Measure Turbo Mode Effect ....................................................................................... 184 Energy Efficient Turbo ................................................................................................. 186 Use vSphere Host Power Management .................................................................. 187 Rethink Your Power Management Strategy ......................................................... 193 MEMORY RESOURCES ................................................................................ 195 PROLOGUE ..................................................................................................... 197 MEMORY ARCHITECTURE ......................................................................... 199 Dual In-Line Memory Module .................................................................................... 200 Memory Organization ................................................................................................... 201 Registered and Unregistered DIMM ........................................................................207 Load-Reduced DIMM .....................................................................................................209 Forward look: 3DS/TSV RDIMMs............................................................................... 213 Memory Protection ........................................................................................................ 214 MEMORY SUBSYSTEM BANDWIDTH ...................................................... 219 DIMM Bandwidth ...........................................................................................................220 Effective Memory Bandwidth .................................................................................... 222 Memory Region ............................................................................................................... 223 DIMM Per Channel Configuration ............................................................................224 DDR4 Bandwidth and CAS Timings ......................................................................... 229 Memory Module Installation Guidelines ................................................................ 230 MEMORY TOPOLOGY DESIGN ................................................................... 231 Memory Speed................................................................................................................. 231 CPU Selection .................................................................................................................. 232 Memory Channel ............................................................................................................ 233 Memory Capacity ........................................................................................................... 234 DIMM Type ....................................................................................................................... 234 Ranks Per DIMM ............................................................................................................. 235 DIMMs Per Channel ....................................................................................................... 236 Unbalanced NUMA Configuration ............................................................................ 239 Unbalanced NUMA Node Configuration ............................................................... 240 Mixed DIMM Capacity Configuration ...................................................................... 241 VMKERNEL MEMORY MANAGEMENT ................................................... 243 Large Pages ..................................................................................................................... 244 Virtual Memory and Virtualized Memory .............................................................. 245 Translation Lookaside Buffer ..................................................................................... 250 TLB MISS ........................................................................................................................... 253 Host Overhead Memory ............................................................................................... 254 VM Overhead Memory ..................................................................................................260 VM Level Resource Allocation Settings ................................................................. 262 Reservation ...................................................................................................................... 265 Shares ................................................................................................................................ 267 Limit ................................................................................................................................... 268 Memory Idle Tax .............................................................................................................269 MEMORY RECLAMATION TECHNIQUES ................................................ 271 MinFree ............................................................................................................................. 272 Transparent Page Sharing .......................................................................................... 277 Memory Ballooning ....................................................................................................... 285 Memory Compression .................................................................................................. 289 Memory Swapping ......................................................................................................... 292 Alternative Swap file location ................................................................................... 297 Host Swap Cache ............................................................................................................ 298 Block Execution .............................................................................................................. 304 STORAGE RESOURCES ................................................................................ 305 PROLOGUE ..................................................................................................... 307 HOST-LOCAL STORAGE ARCHITECTURE ............................................... 309 Transport Interconnect ................................................................................................. 311 Transport Protocols ........................................................................................................318 Form Factor ..................................................................................................................... 324 Selection Criteria ........................................................................................................... 324 Driver .................................................................................................................................. 328 NON-VOLATILE MEMORY ARCHITECTURE .......................................... 331 Flash Technology Architecture ................................................................................ 332 Device Endurance .......................................................................................................... 344 Power Loss Data Protection Features ..................................................................... 347 3D XPoint .......................................................................................................................... 348 Near-Future NVM Solutions ....................................................................................... 355 vSAN STORAGE ARCHITECTURE ............................................................. 361 Scaling Storage Platforms........................................................................................... 362 vSAN 6.6 Requirements ................................................................................................371 vSAN Hardware Compatibility List .......................................................................... 378 vSAN Storage Policies .................................................................................................. 384 vSAN Design Considerations ..................................................................................... 387 CPU Interoperability ..................................................................................................... 404 Use vSAN Observer to Monitor vSAN...................................................................... 407 CORE STORAGE ............................................................................................ 408 Storage I/O Queues ........................................................................................................ 408 Storage Performance Metrics .................................................................................... 415 Pluggable Storage Architecture ................................................................................ 417 VMFS .................................................................................................................................. 419 VIRTUAL MACHINE STORAGE ..................................................................428 Virtual Storage Adapters ............................................................................................. 428 Virtual Disk Types ......................................................................................................... 433 NETWORK RESOURCES .............................................................................. 436 PROLOGUE ..................................................................................................... 438 HOST NETWORK ARCHITECTURE .......................................................... 440 Physical NIC ..................................................................................................................... 441 Form Factor ..................................................................................................................... 442 PCI Express...................................................................................................................... 443 Converged Network Adapters ................................................................................... 445 pNIC Layout ..................................................................................................................... 446 List pNIC Layout ............................................................................................................ 447 Drivers ............................................................................................................................... 448 Offloading ......................................................................................................................... 455 Network Overlay ............................................................................................................ 466 pNIC Features .................................................................................................................. 471 pNIC Support .................................................................................................................... 483 Jumbo Frames.................................................................................................................487 VMKERNEL NETWORK .............................................................................. 494 Netpoll Thread Scaling ................................................................................................ 497 Tx Thread Scaling ......................................................................................................... 499 Tuning Considerations ................................................................................................. 502 VIRTUAL MACHINE NETWORK ................................................................ 522 Virtual Network Adapters ........................................................................................... 522 DirectPath I/O for Networking ................................................................................... 528 Single Root I/O Virtualization .................................................................................... 529 Virtual RDMA ................................................................................................................... 536 vSWITCH UPLINKS ....................................................................................... 540 Link Aggregation Group ............................................................................................... 541 IP-HASH ............................................................................................................................ 542 Load Based Teaming ..................................................................................................... 545 HOST EVICTION ............................................................................................ 550 Designing your vMotion Network ............................................................................ 551 Multi-NIC setup ............................................................................................................... 552 Failover Order and Load Balancing policy ............................................................ 552 DISTRIBUTED STORAGE NETWORK ........................................................ 556 REFERENCES ................................................................................................. 560 P1: CPU Resources .......................................................................................................... 561 P2: Memory Resources ................................................................................................. 562 P3: Storage Resources ................................................................................................... 564 P4: Network Resources ................................................................................................. 564 ACKNOWLEDGEMENTS If you want to go fast go alone. If you want to go far go together. This African proverb resonated strongly with us. Although we have spent countless of hours of writing, editing, rewriting, deleting and rewriting again, this book wouldn't be in its current shape with the help of the following people. We would like to thank Kit Colbert for providing us an amazing foreword. Pat Gelsinger, Carl Waldspurger, Christos Karamanolis and Andrew Lambeth for sharing their thoughts on the state of the industry and how new and future services require a refocus on host design. We want to extend our gratitude towards Myles Gray, Marco van Baggum, Duncan Epping, Rutger Kosters and Anthony Spiteri for reviewing the book from an administrator and architect point of view. Jane Rimmer did her utmost best to educate us on the finesse of the English grammar. Despite the lab providing us immense value, presenting us a test bed to create and destroy many workloads, we stood on the shoulders of many giants by distilling their research work and academic papers. Although reverse engineering helps you grasp the finer details of the product we would extend our gratitude to Chris Gianos (Intel) and VMware employees; Valentin Bondzio, Haoqiang Zheng, Richard Lu, Duncan Epping, Cormac Hogan, Paudie O'Riordan and Pete Koehler to help us stay honest and review the content from a technical perspective. External to VMware, Ed van Hout, Joop Carels and Rosa Martinez Perallon for providing ideas on various network related content and help with test setups. And last but not least, thanks for supporting this book within VMware: Charu Chaubal, Kristine Anderson, Michael Adams and Mark Lohmeyer. 9