AUDITORY USER INTERFACES Toward the Speaking Computer AUDITORY USER INTERFACES Toward the Speaking Computer by T. V. Raman Adobe Systems Incorporated SPRINGER-SCIENCE+BUSINESS MEDIA, LLC ISBN 978-1-4613-7855-6 ISBN 978-1-4615-6225-2 (eBook) DOI 10.1007/978-1-4615-6225-2 Library of Congress Cataloging-in-Publication Data A C.I.P. Catalogue record for this book is available from the Library of Congress. Copyright © 1997 Springer Science+Business Media New York Originally published by Kluwer Academic Publishers in 1997 Softcover reprint ofthe hardcover Ist edition 1997 AII rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, mechanical, photo copying, recording, or otherwise, without the prior written permis sion of the publisher, Springer-Science+Business Media, LLC. Printed on acid-free pap er. I To my guiding eyes~ ASTER i Contents List of Figures ix List of Tables xiii Foreword xv Preface xix Acknowledgements xxi 1. SPEECH-ENABLED APPLICATIONS 1.1 Introduction 1 1.2 What Is UI? 2 1.3 Alternative Modes of Interaction 4 1.4 Retrofitting Spoken Interaction 5 1.5 The Speech-enabling Approach 7 1.6 Separating Computation From User Interface 9 2. NUTS AND BOLTS OF AUDITORY INTERFACES 11 2.1 Introduction 11 2.2 Speech Synthesis 12 2.3 Speech Recognition 18 2.4 Digital Signal Processing Techniques 23 2.5 Auditory Displays And Audio Formatting 28 2.6 Interactive User Interface Development 32 3. THE AUDIO DESKTOP 39 3.1 Introduction 39 3.2 The Visual Desktop 40 3.3 Conversational Gestures 44 3.4 Choosing Abstractions For The Audio Desktop 49 4. CONCRETE IMPLEMENTATION OF AN AUDIO DESKTOP 55 4.1 Introduction 55 4.2 Basic Services For Speech-enabling The Desktop 56 4.3 The Emacspeak Desktop 64 vii viii AUDITORY USER INTERFACES (AUI) 4.4 Speech-enabled Editing Tools 72 4.5 Structured Editing And Templates 77 4.6 Browsing Structured Information 86 4.7 Information Management On The Audio Desktop 93 4.8 Speech-enabled Messaging Tools 95 4.9 Editing Program Source 102 4.10 Software Development Environment 107 4.11 Technique Used To Speech-enable Emacs 113 4.12 Thanking The Emacs Community 118 5. SPEECH-ENABLING THE WWW 121 5.1 Introduction 121 5.2 Aural Information Access 122 5.3 Web Surfing Without A Monitor 125 Bibliography 133 Index 137 List of Figures 1.1 Computing applications typically consist of obtaining user input, com- puting on this information and finally displaying the results. The first and third phase in this process constitute the user interface. As can be seen, it is possible to separate the user interface from the computational phase. 3 1.2 Calendars are displayed visually using a two dimensional layout that makes it easy to see the underlying structure. The calendar display consists of a set of characters on the screen; but the meaning of this display is as much in its visual layout as in the characters themselves. Merely speaking the text fails to convey meaning. We can see that January 1, 2000 is a Saturday; this information is missing when the visual display is spoken. 7 2.1 Sub-components of recorded prompts used by an IVR system at a bank. Different prompts can be generated by concatenating appropriate components. 14 2.2 Phonemes in American English. The various vowels and consonants making up standard American English are shown using a two-letter notation. Each phoneme is shown along with a word containing that phoneme. 15 2.3 Textual description of a nested exponent. Notice that when reading the prose making up this description, it is very difficult to perceive the underlying structure of the mathematical expression. 18 2.4 A call management system using word spotting. Users can express the same command in several ways. The recognition system looks for key phrases that determine the user command, thereby allowing for a flexible system. 20 ix x AUDITORY USER INTERFACES (AU!) 2.5 Coarticulatory effects in continuous speech. Co articulatory effects (or the lack there of) are often a problem when trying to synthesize natural sounding speech. Not surprisingly, the presence of these same effects in human speech make the computer's task of recognizing continuous speech even harder. 21 2.6 Using spatial audio to encode information about incoming email. Au ditory cues indicate the arrival of new mail. These auditory cues encode additional information such as urgency of the message using spatial audio. 26 3.1 Visual realization of conversational gestures -the building blocks for dialogues. User interface design tries to bridge the impedance mis match in man-machine communication by inventing a basic set of con versational gestures that can be effectively generated and interpreted by both man and machine. 45 4.1 The Emacspeak desktop consists of a set of active buffer objects. This display shows a subset of currently active buffers on my desktop. 66 4.2 A sample directory listing. The visual interface exploits vertical align- ment to implicitly encode the meaning of each field in the listing. 69 4.3 A listing of running processes. The task manager helps in tracking system resources. Processes can be killed or suspended from the task manager. 72 4.4 Commands available while searching. A set of highly context-specific conversational gestures. 75 4.5 Outline view of this section. It can be used to move quickly to different logical components of the document. 78 4.6 Result of folding the lexical analyzer in J\l)T~. This is a document consisting of over 2, 000 lines. Folding helps in organizing the code, obtaining quick overviews, as well as in efficient navigation. 80 4.7 Sample collection of dynamic macros available when editing C source. Standard C constructs can be generated with a few gestures. 82 4.8 A sample C program. It can be created with a few gestures when using dynamic macros. 83 4.9 A sample HTML page. Template-based authoring makes creating such documents easy. 84 4.10 Visual display of a structured data record. The data record is visually formatted to display each field name along with its value. 85 4.11 An expense report. Semantics of the various fields in each record is implicitly encoded in the visual layout. 86 4.12 Tracking an investment portfolio. Modifying entries can cause com- plex changes to the rest of the document. 87 LIST OF FIGURES Xl 4.13 A train schedule. We typically look for the information we want, rather than reading the entire timetable. 91 4.14 Commands in table browsing mode. The interface enables the user to locate the desired item of information without having to read the entire table. 92 4.15 A well-formatted display of the message headers presents a succinct overview of an email message in the visual interface. Speaking this visual display does not produce a pleasant spoken interface -the spoken summary needs to be composed directly from the underlying information making up the visual display. 98 4.16 Newsgroups with unread articles are displayed in a *Group* buffer. This buffer provides special commands for operating on newsgroups. The visual interface shows the name of the group preceded by the number of unread articles. 100 4.17 Unread articles are displayed in buffer *Group Summary*. This buffer is augmented with special commands for reading and responding to news postings. The visually formatted output succinctly conveys arti- cle attributes such as author and subject. 101 4.18 More than one opening delimiter can appear on a line. When typing the closing delimiter, Emacspeak speaks the line containing the matching delimiter. The spoken feedback is designed to accurately indicate which of the several open delimiters is being matched. 106 4.19 An example of comparing different versions of a file. Visual layout exploits changes in fonts to set apart the two versions. The reader's attention is drawn to specific differences by visual highlighting -here, specific differences are shown in a bold font. Visual interaction relies on the eye's ability to quickly navigate a two dimensional display. Directly speaking such displays is both tedious and unproductive. 110 4.20 Browsing the Java Development Kit (JDK 1.1) using a rich visual interface. Understanding large object oriented systems requires rich browsing tools. Emacspeak speech-enables a powerful object oriented browser to provide a pleasant software development environment. 114 4.21 Emacspeak is implemented as a series of modular layers. Low-level layers provide device-specific interfaces. Core services are imple mented on a device-independent layer. Application-specific extensions rely on these core services. 115 4.22 Advice is a powerful technique for extending functionality of pre existing functions without modifying their source code. Here, we show the calling sequence for a function f that has before, around, and after advice defined. 117 xii AUDITORY USER INTERFACES (AUI) 4.23 Example of advising a built-in Emacs command to speak. Here, com- mand next-line is speech-enabled via an after advice that causes the current line to be spoken after every user invocation of this command. 117 5.1 HTML pages on the WWW of the 1990's abound in presentational markup. What does red text on a monochrome display mean? What does it mean to (er) blink aurally? 123 5.2 A sample aural style sheet fragment for producing audio formatted Webformation. Audio formatting conveys document structure implic- itly in the aural rendering, allowing the listener to focus on the infor- mation content. 125 5.3 The HTML 3.2 specification fails to separate the underlying conversa- tional gesture from its visual realization even more dramatically than GU! toolkits. In this example, it is impossible to decipher from the markup that the current dialogue expects the user to enter a name and age -in HTML 3.2, there is no association between an edit field and its label. 129 5.4 The AltaV ista main page. This page presents a search dialogue using a visual interface. Emacspeak presents a speech-enabled version of this dialogue that is derived from the underlying HTML. 130