Automatically Generating High-Quality User Interfaces for Appliances Jeffrey Nichols December 2006 CMU-HCII-06-109 Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Pittsburgh, Pennsylvania 15213 Thesis Committee: Brad A. Myers, Chair Scott E. Hudson John Zimmerman Dan R. Olsen, Jr., Brigham Young University Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy Copyright © 2006 Jeffrey Nichols. All rights reserved. This research was supported in part by the National Science Foundation (through the author’s Graduate Re- search Fellowship and grants IIS-0117658 and IIS-0534349), Microsoft, General Motors, the Pittsburgh Digital Greenhouse, and Intel. Equipment supporting this research was generously donated by Mitsubishi Elec- tric Research Laboratories, VividLogic, Lantronix, Lutron, IBM Canada, Symbol Technologies, Hewlett- Packard, and Lucent. The views and conclusions contained herein are those of the author and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of any sponsoring party or the U.S. Government. Keywords: Automatic interface generation, aggregate user interfaces, handheld computers, personal digital assistants, mobile phones, home theater, appliances, personal universal con- troller (PUC), user interface description languages (UIDLs), remote controls, multi-modal interfaces, speech recognition, Smart Templates, user interface consistency, personal consis- tency, familiarity ii Automatically Generating High-Quality User Interfaces for Appliances Jeffrey Nichols iii iv Abstract In this dissertation, I show that many appliance usability problems can be addressed by mov- ing the user interface from the appliance to a handheld device that the user is already carrying, such as a Personal Digital Assistant (PDA) or mobile phone. This approach, called the Personal Universal Controller (PUC), takes advantage of the increasing pervasiveness of wireless communication technologies that will allow handheld devices to communicate di- rectly with appliances. Automatic generation of the appliance user interface allows the PUC to create interfaces that are customized to the platform of the controller device, the user’s previous experience, and all the appliances that are present in the user’s current environment. This dissertation makes several contributions to the state of the art in automatic interface generation: • Automatic generation that makes use of dependency information to determine better structure for the generated user interfaces. • The general Smart Templates technique for incorporating domain-specific design con- ventions into an appliance specification and automatically rendering the conventions appropriately on different platforms and in different interface modalities • Algorithms that apply knowledge of similarity between specifications and interfaces the user has previously seen to generate new interfaces that are personally consistent • Algorithms that use a model of the content flow between appliances to generate task- based interfaces that combine functionality from multiple appliances An evaluation of the PUC system compared the automatically generated interfaces for two all-in-one printers with the manufacturer’s interfaces for the same two appliances and found that users of the automatically generated interfaces were twice as fast and four times as suc- cessful for both common and complex tasks. The evaluation also shows that the PUC’s consistency features allow users to be twice as fast when using a new appliance that is similar to an appliance they have previously encountered. This evaluation is the first known user study of automatically generated interfaces compared to human designs, and it shows that automatic generation of user interfaces for end users is now viable for interactive systems. v vi For my parents vii viii Table of Contents Abstract.................................................................................................................................v List of Figures...................................................................................................................xvii List of Tables.....................................................................................................................xxv Acknowledgements.........................................................................................................xxvii 1 Introduction 1 1.1 The Personal Universal Controller.....................................................................7 1.2 Outside the Scope............................................................................................13 1.3 Contributions..................................................................................................14 1.4 Dissertation Overview......................................................................................15 2 Related Work 17 2.1 Control of Appliances......................................................................................18 2.1.1 Commercial Products.............................................................................18 2.1.2 Commercial Standards...........................................................................20 2.1.2.1 INCITS/V2 Standard.............................................................20 2.1.2.2 Universal Plug and Play..........................................................21 2.1.2.3 Digital Living Network Alliance..............................................22 2.1.2.4 Home Audio-Video Interoperability.........................................22 2.1.2.5 JINI.......................................................................................23 2.1.2.6 OSGi.....................................................................................23 2.1.3 Research Systems....................................................................................23 2.1.3.1 Universal Interactor................................................................23 2.1.3.2 IBM Universal Information Appliance.....................................24 2.1.3.3 ICrafter..................................................................................24 2.1.3.4 Xweb......................................................................................25 2.1.3.5 Ubiquitous Interactor..............................................................25 ix 2.1.3.6 Analyses of Actual Remote Control Usage..................................26 2.1.3.7 DiamondHelp.........................................................................26 2.1.3.8 Roadie....................................................................................26 2.2 Automatic & Guided User Interface Design.....................................................27 2.2.1 Early Model-Based Systems.....................................................................28 2.2.1.1 Mickey....................................................................................28 2.2.1.2 Jade........................................................................................29 2.2.1.3 UIDE.....................................................................................29 2.2.1.4 Humanoid..............................................................................30 2.2.1.5 Mastermind............................................................................30 2.2.1.6 ITS........................................................................................30 2.2.1.7 TRIDENT.............................................................................31 2.2.2 Model-Based Systems for Very Large Interfaces and Platform Independence31 2.2.2.1 Mobi-D..................................................................................32 2.2.2.2 ConcurTaskTrees....................................................................32 2.2.2.3 XIML.....................................................................................33 2.2.2.4 IBM PIMA and MDAT.........................................................33 2.2.2.5 UIML and TIDE...................................................................33 2.2.2.6 TERESA................................................................................34 2.2.2.7 USIXML................................................................................34 2.2.2.8 XAML and XUL.....................................................................34 2.2.2.9 SUPPLE................................................................................35 2.3 Aggregate User Interfaces.................................................................................36 3 Preliminary User Studies 39 3.1 Hand-Designed User Interfaces........................................................................39 3.2 User Studies.....................................................................................................42 3.2.1 Procedure..............................................................................................42 3.2.2 Evaluation............................................................................................42 3.3 Study #1..........................................................................................................43 x