Live Migration of User Environments Across Wide Area Networks Cristian Zamfir Submitted in fulfillment of the requirements for the Degree of Master by Research. Department of Computing Science Faculty of Information and Mathematical Sciences University of Glasgow October 2008 2 Abstract A complex challenge in mobile computing is to allow the user to migrate her highly customised environment while moving to a different location and to continue work without interruption. I motivate why this is a highly desirable capability and conduct a survey of the current approaches towards this goal and explain their limitations. I then propose a new architecture to support user mobility by live migration of a user’s operating system instance over the network. Previous work includes the Collective and Internet Suspend/Resume projects that have addressed migration of a user’s environment by suspending the running state and resuming it at a later time. In contrast to previous work, this work addresses live migration of a user’s operating system instance across wide area links. Live migration is done by performing most of the migration while the operating system is still running, achieving very little downtime and preserving all network connectivity. I developed an initial proof of concept of this solution. It relies on migrating whole operating systems using the Xen virtual machine and provides a way to perform live migration of persistent storage as well as the network connections across subnets. These challenges have not been addressed previously in this scenario. In a virtual machine environment, persistent storage is provided by virtual block devices. The architecture supports decentralized virtual block device replication across wide area network links, as well as migrating network connection across subnetworks using the Host Identity Protocol. The proposed architecture is compared against existing solutions and an initial performance evaluation of the prototype implementation is presented, showing that such a solution is a promising step towards true seamless mobility of fully fledged computing environments. Acknowledgments I would like to acknowledge the help and advice I received from the following people. My thesis supervisors Dr Peter Dickman and Dr Colin Perkins for the invaluable guidance, advice, and encouragement I received throughout the project and for the help in improving this thesis. Professor Joe Sventek for the feedback, help and support I received during this project. The Xen and DRBD communities for answering my questions and for the feedback I received, especially Keir Fraser for the quick fixes to network device migration in Xen, Lars Ellenberg for the help provided in understanding the technical aspects of block device migration in DRBD and Octavian Purdila for the help regarding udev. The technical staff, especially Douglas MacFarlane and Naveed Khan for the help in setting up the network testbed. Students Ross McIlroy, Stephen Strowes, Mark Shannon, Athanasios Pavlopoulos and Dora Galvez-Cruz for the early feedback during the initial steps of my work. Students Silviu Andrica and Paul Marinescu for proof-reading the thesis. My parents for their continuous support, without which this work wouldn’t have been possible. Alexandra Covaci for the support and encouragement during the writing up of the thesis. 4 Contents 1 Introduction 10 1.1 Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.2 Discussion and Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.3 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.4 Thesis Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.5 Structure of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2 Background 18 2.1 Virtual Machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.1.1 The Xen Hypervisor . . . . . . . . . . . . . . . . . . . . . . . . 21 2.2 I/O Virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.2.1 The Xen I/O Subsystem . . . . . . . . . . . . . . . . . . . . . . 23 2.2.2 Soft Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.3 Mobility Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 5 2.3.1 Survival of TCP Connections for Long Periods of Disconnection . 27 2.4 Storage Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.4.1 Mirroring Virtual Disks . . . . . . . . . . . . . . . . . . . . . . . 33 2.5 Migration of Virtual Machines and Environments . . . . . . . . . . . . . 36 2.5.1 Virtual Machine Live Migration . . . . . . . . . . . . . . . . . . 36 2.5.2 Migration of Virtual Environments . . . . . . . . . . . . . . . . . 37 2.5.3 Persistent Storage Management . . . . . . . . . . . . . . . . . . 44 2.5.4 Xen Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 2.6 Quantifying Interactive User Experience . . . . . . . . . . . . . . . . . . 47 2.7 Live Migration of User Environments . . . . . . . . . . . . . . . . . . . 48 2.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3 Architecture 50 3.1 Discussion of the Factors that Influence the Total Live Migration Time . . 51 3.2 General Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 3.3 Specific Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.3.1 Virtual Block Device Live Migration . . . . . . . . . . . . . . . 55 3.3.2 Reducing the Impact of the Migration on Network Traffic . . . . 57 3.3.3 Migrating Network Connections across Subnets . . . . . . . . . . 58 3.3.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 6 4 Implementation 62 4.1 Persistent Storage Migration . . . . . . . . . . . . . . . . . . . . . . . . 63 4.2 Live Migration of Virtual Block Devices . . . . . . . . . . . . . . . . . . 64 4.3 Migration of Network Connections . . . . . . . . . . . . . . . . . . . . . 67 4.4 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 4.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 5 Performance Evaluation 73 5.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 5.2 Network Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 5.3 Disk I/O Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 5.4 Wrap-up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 6 Future Work 86 6.1 User Activity Traces . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 6.2 Asynchronous Replication . . . . . . . . . . . . . . . . . . . . . . . . . 87 6.3 Estimating the Impact of Migration on User Experience . . . . . . . . . . 87 6.4 Live Migration across Hypervisors for Mobile Devices . . . . . . . . . . 88 7 Summary and Conclusions 90 7 List of Figures 1.1 Scenario: Professor X’s virtual environment is live migrated between vari- ous physical hosts around the world such as her office PC or her in-car PC. Her system is not suspended during the commutes so that network activity such as attending a video conference is not affected. . . . . . . . . . . . . 13 2.1 Architecture of the Xen Virtual Machine. . . . . . . . . . . . . . . . . . 21 2.2 Host Identity Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.1 Architecture for live migration of user environments . . . . . . . . . . . . 53 4.1 Currently, live migration of virtual block devices can be done using SAN. 63 4.2 Live migration of virtual block devices. . . . . . . . . . . . . . . . . . . 65 4.3 Migration of network connections using HIP. . . . . . . . . . . . . . . . 68 5.1 Performance Evaluation Setup . . . . . . . . . . . . . . . . . . . . . . . 74 5.2 Live Migration across 100 Mbps Ethernet of a VM running an Iperf client using the unmodified Xen live migration (no storage migration was per- formed). The storage was mounted from a remote server using the ATA over Ethernet protocol. Migration occurs in the same subnetwork. No additional network delays were introduced. . . . . . . . . . . . . . . . . 76 8 5.3 Live Migration across 100 Mbps Ethernet of a VM running Iperf in client mode. The experiment is done using the VBD migration prototype. No additional network delays were introduced. . . . . . . . . . . . . . . . . 77 5.4 TCP throughput measured on the server during migration across subnet- works using HIP. VBD migration is also performed. No additional net- work delays were introduced. . . . . . . . . . . . . . . . . . . . . . . . . 79 5.5 TCP throughput measured on the client during migration across subnet- works using HIP. VBD migration is also performed. No additional net- work delays were introduced. . . . . . . . . . . . . . . . . . . . . . . . . 80 5.6 I/O performance of sequential input during virtual block device migration over a 100Mbps link measured with the Bonnie++ benchmark. . . . . . . 82 5.7 I/O performance of sequential output during virtual block device migra- tion over a 100Mbps link measured with the Bonnie++ benchmark. . . . . 83 9 Chapter 1 Introduction Coping with user mobility across pervasive hardware while maintaining access to a highly customized environment is a challenging task and today’s computing environments do not provide adequate support for it. The operating system, installed software and user data take up several gigabytes of storage and are tightly coupled with the hardware. Common solutions for user mobility include mounting user profiles stored on distributed file systems, roaming profiles, remote connections to the user’s machine, process migration, carrying a heavy and fragile laptop or using a mobile device with limited computing capabilities. All these solutions have drawbacks that prevent a user from resuming work without interruption across various computers. For instance, remote connections to a user’s machine are prone to network disconnections and are sensitive to the network delay. Mobile devices such as PDAs typically have a small computing power and a limited interface. Laptops are fragile devices that can compromise user data or be stolen. Ideally, a user would like to use the same settings and data as well as the same software packages everywhere. This would allow the user to use the software and settings that she is most accustomed to. Moreover, the same data should be available at every location where the user is using the computing environment (i.e. at home and at the work place). Unfortunately, there are many hindrances to accomplishing this ideal. Firstly, even though the user can take advantage of many machines, typically the user environment on each of them is significantly different because the installed software and the user settings are different. Moreover, the user has to take care of synchronizing the data that she modified on one machine so that it is available on the others. Network connections are 10