An Inertial Measurement Unit for User Interfaces by Ari Yosef Benbasat B.A.Sc., University of British Columbia (1998) Submitted to the Program in Media Arts and Sciences, School of Architecture and Planning, in partial fulfillment of the requirements for the degree of Master of Science in Media Arts and Sciences at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY September 2000 (cid:13)c Massachusetts Institute of Technology 2000. All rights reserved. Author Program in Media Arts and Sciences September 8, 2000 Certified by Joseph A. Paradiso Principal Research Scientist MIT Media Laboratory Thesis Supervisor Accepted by Stephen A. Benton Chair, Department Committee on Graduate Students Program in Media Arts and Sciences 2 An Inertial Measurement Unit for User Interfaces by Ari Yosef Benbasat Submitted to the Program in Media Arts and Sciences, School of Architecture and Planning, on September 8, 2000, in partial fulfillment of the requirements for the degree of Master of Science in Media Arts and Sciences Abstract Inertial measurement components, which sense either acceleration or angular rate, are be- ing embedded into common user interface devices more frequently as their cost continues to drop dramatically. These devices hold a number of advantages over other sensing technolo- gies: they measure relevant parameters for human interfaces and can easily be embedded into wireless, mobile platforms. The work in this dissertation demonstrates that inertial measurement can be used to acquire rich data about human gestures, that we can derive efficient algorithms for using this data in gesture recognition, and that the concept of a parameterized atomic gesture recognition has merit. Further we show that a framework combining these three levels of description can be easily used by designers to create robust applications. A wireless six degree-of-freedom inertial measurement unit (IMU), with a cubical form factor (1.25 inches on a side) was constructed to collect the data, providing updates at 15 ms intervals. This data is analyzed for periods of activity using a windowed variance algorithm, whose thresholds can be set analytically. These segments are then examined by the gesture recognition algorithms, which are applied on an axis-by-axis basis to the data. The recognized gestures are considered atomic (i.e. cannot be decomposed) and are parameterized in terms of magnitude and duration. Given these atomic gestures, a simple scripting language is developed to allow designers to combine them into full gestures of interest. It allows matching of recognized atomic gestures to prototypes based on their type, parameters and time of occurrence. Because our goal is to eventually create stand-alone devices,the algorithms designed for this framework have both low algorithmic complexity and low latency, at the price of a small loss in generality. To demonstrate this system, the gesture recognition portion of (void*): A Cast of Characters, an installation which used a pair of hand-held IMUs to capture gestural inputs, was implemented using this framework. This version ran much faster than the original version (based on Hidden Markov Models), used less processing power, and performed at least as well. Thesis Supervisor: Joseph A. Paradiso Title: Principal Research Scientist, MIT Media Laboratory 3 4 An Inertial Measurement Unit for User Interfaces by Ari Yosef Benbasat The following people served as readers for this thesis: Thesis Reader Bruce M. Blumberg Assistant Professor of Media Arts and Sciences Asahi Broadcasting Corporation Career Development Professor of Media Arts and Sciences Thesis Reader Paul A. DeBitetto Program Manager Special Operations and Land Robotics Group Charles Stark Draper Laboratory 5 6 Acknowledgments To Joe Paradiso, who brought me to the Lab when I knew nothing, and then provided me with his enthusiastic suggestions, directions and advice to support my growth. If I am one day a good engineer, much of it will be Joe’s doing. To Bruce Blumberg and Paul DeBitteto, my gracious readers, for their help and will- ingness to operate under deadline. Also, many thanks to Bruce for asking me to work on (void*). To all those who helped me along: Matt Berlin and Jesse Gray, who coded up the scripting system at the last moment and contributed to its structure; Andy Wilson for wonderful advice on gesture recognition; Michael Patrick Johnson for his patience in helping me with quaternion math; Marc Downie for his patience in helping me with virtually everything else; Ari Adler, who did some early work on this project; Jacky Mallett, my partner in crime in late-summer thesis preparation; and Pamela Mukerji, who wrote some of the embedded code and helped hunt down references and figures. To Ari Adler and Ara Knaian, wonderful officemates who made the rough times at the Lab bearable and the good times better. To the Responsive Environments Group, my good friends and home away from home. To the Synthetic Characters, for use of their code base and their friendship. It was a joy to work on (void*) with them. To the Natural Sciences and Engineering Research Council, for financial support. Andfinally, tomy parents, whohaveneversparedanytime, effortorloveifitwouldmake my life in any way better. 7 8 Contents Abstract 3 List of Figures 13 List of Tables 15 1 Introduction 17 1.1 Historical Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 1.1.1 Inertial Technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 1.1.2 Gesture-Based Systems . . . . . . . . . . . . . . . . . . . . . . . . . 19 1.2 Prior Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 1.3 Project Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2 Hardware Systems 25 2.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.2 IMU High Level Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.2.1 Core functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.2.2 Mechanical Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.2.3 Subsidiary systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.3 IMU Component Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.3.1 Inertial Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.3.2 Processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.3.3 Radio Frequency Transmitters/Receivers. . . . . . . . . . . . . . . . 32 2.3.4 Other Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 2.4 IMU Low Level Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 2.4.1 Core functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 2.4.2 Mechanical Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 2.4.3 Subsidiary systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 2.5 Microcontroller Embedded Code . . . . . . . . . . . . . . . . . . . . . . . . 39 2.5.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.5.2 RF Transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 2.5.3 Capacitive Signaling . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 2.6 Receiver Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 2.7 Receiver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 9 3 Sample Data Stream 45 4 Analysis 49 4.1 Kalman Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 4.1.1 Principles and Algorithms . . . . . . . . . . . . . . . . . . . . . . . . 50 4.1.2 Limits of Inertial Tracking . . . . . . . . . . . . . . . . . . . . . . . . 52 4.2 Frequency Space Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 4.2.1 Low Pass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 4.2.2 High Pass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 4.3 Activity Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 5 Gesture Recognition 63 5.1 Key Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 5.1.1 Gesture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 5.1.2 The Parameterization . . . . . . . . . . . . . . . . . . . . . . . . . . 65 5.2 State-Based Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 5.3 Single-Axis Gesture Recognition . . . . . . . . . . . . . . . . . . . . . . . . 68 5.3.1 Pros/Cons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 5.3.2 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 5.3.3 Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 5.4 Entropic Limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 6 Output Scripting 79 6.1 Scripting System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 6.1.1 High-Level Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 6.1.2 Low-Level Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 6.2 Sample Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 7 Sample Application 85 7.1 Solvable Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 7.2 (void*): A Cast of Characters . . . . . . . . . . . . . . . . . . . . . . . . . 86 7.2.1 Summary of Installation . . . . . . . . . . . . . . . . . . . . . . . . . 87 7.2.2 Then . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 7.2.3 Now . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 7.2.4 Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 8 Conclusions 93 8.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 8.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 8.3 Future Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 A Abbreviations and Symbols 99 B Glossary 101 C Schematics, PCB Layouts and Drawings 103 10