DOCUMENT RESUME IR 058 812 ED 482 136 Sauer, Sebastian; Gobel, Stefan AUTHOR Focus Your Young Visitors: Kids Innovation--Fundamental Changes in TITLE Digital Edutainment. 2003-00-00 PUB DATE 16p.; In: Museums and the Web 2003: Selected Papers from an NOTE International Conference (7th, Charlotte, NC, March 19-22, 2003); see IR 058 801. For full text: http://www.archimuse.com/mw2003/ AVAILABLE FROM papers/sauer/sauer.html/. Speeches/Meeting Papers (150) Descriptive (141) Reports PUB TYPE EDRS Price MF01/PC01 Plus Postage. EDRS PRICE Children; *Computer Games; Computer Interfaces; Computer System DESCRIPTORS Design; *Exhibits; Immersion Programs; Interdisciplinary Approach; Multimedia Materials; *Museums; Story Telling ABSTRACT With regard to the acceptance of human-computer interfaces, immersion represents one of the most important methods for attracting young visitors into museum exhibitions. Exciting and diversely presented content as well as intuitive, natural and human-like interfaces are indispensable to bind users to an interactive system with real and digital parts. In order to overcome the obstacles to attracting the young, the authors have taken an interdisciplinary approach to the fields of multimedia, media design,TV/movie or literature, and especially myths, to develop narrative "edutainment" applications for children. Similar to Hollywood, production content and dramaturgic scripts are generated by authoring tools and transmitted by multimodal interfaces enhanced by natural conversation forms. These concepts are enhanced by interaction design principles, methods and appliances; such as Kids Innovation or Momuna.This paper describes both existing technologies and new innovative methods and concepts of interactive digital storytelling and user interaction design to establish immersive edutainment applications and appliances using multimodal user interfaces. These approaches merging into Dino-Hunter as innovative game-oriented mobile edutainment appliances are discussed within the envi"ronment of mobile application scenarios for museums and their young visitors. (Author) (Contains 18 references.) Includes 13 figures. Reproductions supplied by EDRS are the best that can be made from the original document. PAPERS theWeb Museums and 2003 Focus Your Young Visitors: Kids Innovation Fundamental Changes In Digital Edutainment Sebastian Sauer, ion2s buero fuer interaktion, and Stefan Gäbel, ZGDV e.V. Digital Storytelling, Germany Register Abstract Workshops PERMISSION TO REPRODUCE AND DISSEMINATE THIS MATERIAL HAS Sessions BEEN GRANTED BY With regard to the acceptance of human-computer interfaces, immersion Speakers represents one of the most important methods for attracting young visitors D. Bearman Interactions into museum exhibitions. Exciting and diversely presented content as well as intuitive, natural and human-like interfaces are indispensable to bind Demonstrations users to an interactive system with real and digital parts. Exhibits TO THE EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) Events In order to overcome the obstacles to attracting the young, we have taken 1 an interdisciplinary approach to the fields of multimedia, media design, Best of the Web TV/movie or literature, and especially myths, to develop narrative Key Dates edutainment applications for kids. Similar to Hollywood, production content EDUCATION U.S. DEPARTMENT OF and Improvement Office of Educational Research and dramaturgic scripts are generated by authoring tools and transmitted Charlotte INFORMATION EDUCATIONAL RESOURCES by multimodal interfaces enhanced by natural conversation forms. These CENTER (ERIC) reproduced as concepts are enhanced by interaction design principles, methods and This document has been organization received from the person or appliances; such as Kids Innovation or Momuna. originating it. A&MI to Minor changes have been made improve reproduction quality. This paper describes both existing technologies and new innovative Archives & Museum methods and concepts of interactive digital storytelling and user interaction stated in this Informatics design to establish immersive edutainment applications and appliances Points of view or opinions represent document do not necessarily 158 Lee Avenue using multimodal user interfaces. These approaches merging into Dino- official OERI position or policy. Toronto Ontario Hunter as innovative game-oriented mobile edutainment appliances are M4E 2P3 Canada discussed within the environment of mobile application scenarios for museums and their young visitors. ph: +1 416-691-2516 fx: +1 416-352-6025 Keywords: Kids Innovation, Momuna, Interactive Digital Storytelling, Dino-Hunter, Telebuddy, GEIST, Experience Appliances, Transforming User Interface, Multimodal Interfaces, Digital Information Booth, Edutainment Applications. [email protected] www.archimuse.com Introduction Search The global aims and tasks of museums are not limited to the archival storage and A&MI conservation of cultural and scientific exhibits and artifacts. Rather, the transmission of knowledge, the imprint of cultural identity and the experience of Join our Mailing List. art represent special issues to museums and involved parties in the museum Privacy. scenario. In order to be an active part of cultural life, it is necessary to compete with other actual offers and trend-setting developments in the current information society and to influence these developments. This paper describes practical examples, concepts and methods of innovative digital and interactive edutainment applications enabling especially young visitors to dive into a game- Updated: March 13, 2003 oriented experience and learning environment within museums. Examples for this are integrated multimedia systems of experience appliances consisting of avatar- based information kiosks, mixed reality environments, interactive toy characters and mobile information devices, all based on alternative trend-setting user interfaces providing multimodal and adventure-driven access to the contents of museums. The common scenario and introduced new concept of Kids Innovation is modularly structured and addresses exactly these issues mentioned above. Altogether, Kids Innovation focuses on young people and provides mechanisms and methodologies for fundamental changes in digital edutainment applications. Thus, new ways of interaction and communication between kids and artifacts, between visitors and the museum, or between kids and visitors are introduced. AVAILABLE BEST COPY 0 file://E:\mw2003\papers\sauer\sauer.html 5/27/2003 From the technical point of view, Kids Innovation is enhanced by different principles and mechanisms of interactive digital storytelling and user-centered interaction design methods. Interactive storytelling techniques enable museum institutions to provide exciting stories and interesting presentation modes "beyond the desktop", enabling (young) visitors to dive into a story and get detailed information about artifacts using game-based interfaces. These game- based interfaces are integrated within a mixed reality environment consisting of physical interfaces/props and devices such as sensors, video-tracking, speech recognition or scanners, as well as virtual components such as a 3D-environment with narrative characters talking to the visitors, and multimedia presentations. Subsequently, interactive storytelling techniques improve the usability of digital edutainment applications and enable the creation of exiting, suspenseful and immersive interfaces. From the commercial and marketing-oriented point of view, the use of experience appliances developed by the principles of Digital Storytelling and of Kids Innovation enables museum institutions to get valuable feedback concerning the impact and effect of their own exhibition concepts and the behavior of (young) visitors. Based on these facts and additional case studies, museum institutions are enabled to support schools, classes or any other kind of visitor group within the three major phases of visiting a museum: preparation, execution and post- processing. State-of-the Art User Centred Interaction Design New high-end technology in the field of computers, multimedia systems and telecommunication arises every day and promises a better and easier way to manage human beings' lives. The advantages of such technologies are their nearly never-ending funcitionalities and possibilities. But the crux is, that their complexity is mainly in contrast to intuitive access of their functions. The human being has needs and wants to solve problems without being stressed by having to understand extensive technologies. To let the human being become the centre of the considerations during the development process of technological innovations is the goal of User Centred Interaction Design. Scientists, product and communication designers, computer engineers, psychologists and anthropologists combine their competences to make things usable. Recently, most multimedia systems exclusively have worked with the combination of haptic hardware (solid-) and traditional graphical user interfaces. You find such user interaction components and principles combined in standard Desktop PCs, Personal Digital Assistants, TabletPCs and thousands of software applications. Sometimes auditory components are integrated, but in a very reduced way. A combined solid-/audio- or audio-only interface is associated with devices which are normally developed for physical handicapped people (for example, the blind) or which are used in a special context, where visual attention cannot be paid to the device; for example, while driving a car. Nowadays users get off of their office desktop and dive into a mobile situation. That fact is reflected in new industry interactive and digital products for mobile usage, such as smartphones, Pocket PCs or CarPCs. This is an interesting and important point for the User Interface Developer. The mobile situation and the permanent changes of user surroundings cannot be clearly defined as in a more constant environment; for example, at the desktop in the office. The knowledge of how the five senses of the human being work together and how their reactions to different impressions influence the interaction between the user and a device is more important then ever. This point is very interesting within the museum context because of the combination of real artefacts in interaction with the digital appliances. User Interface combinations and principles show that an assimilation of information can be done parallel by our senses. But every perception channel 3 5/27/2003 file://E: \mw2003 \papers \ sauer\ sauer.html can support only one orientation of our attention. Jef Raskin [13] picked up on this problem and introduces the "locus of attention" in combination with the term "singularity" [10]. If the User is involved in one linear thinking process, can an unforeseen event deflect attention in a new way? The decision is that no second locus of attention arises, but the old one breaks down and the new one takes priority. This fact is very interesting for User Interaction Specialists. It influences especially the design process of human device interfaces for mobile multimedia systems because of the constantly changing environment of the user in the mobile situation. The conclusion is that multimodal Interfaces can provide different access to information and allow different communication modes, but you have to understand the temporary context of the user, his physical and emotional possibilities, the variable environment and the dynamic situation. That means that a static multimodal user interface which interacts with a user by constant modes ignores the changes and cannot be so effective as it could be. (For more on this, see "new concepts..."). Digital Storytelling From the content-related point of view, Digital Storytelling represents a new research discipline contributing to information technology by applying aspects of storytelling and entertainment to digital applications. New authoring methods for multimedia design and application development are currently being explored and implemented by an interdisciplinary team of communication designers and computer scientists. The goal of digital storytelling is to achieve new types of applications that are easily understood by everyone through telling a story [16]. Here, additionally, those new forms of multimodal interaction are integrated. For integrated authoring, three aspects of interactive multimedia play roles in a scenario design description: Graphic-interactive media: These are 2D or 3D graphical elements (including spatial sound sources) which can be placed, designed, and depicted in various displays, as well as manipulated by navigation and pointing techniques (an area already represented by Screen Designers and 3D Designers for virtual reality). Continous media: These include all kinds of time dependent media, such as audio/video streams, animation, and speech. Design criteria are not questions of spatial layout, but of sequencing and timing as well as turn- taking in a conversation. They determine the dramaturgy of an application to a high degree. Physical Media & Modalities: Selection and design of physical-spatial worlds of adequate input and output devices, considering new requirements such as mobility and ubiquitous computing beyond pure screen design. Ergonomic and haptic qualities determine usability, acceptance, and aesthetics. Designers must integrate, design, and evaluate all of these elements during the concept phase of a production. Ultimately, it is the end-user who will evaluate the overall impression of these integrated aspects and determine the application's usability and value. To demonstrate the potential of this integrated design, ZGDV Darmstadt has developed a kiosk system for Siggraph 1999 that gives visitors the impression of stopping at a virtual trade show booth where they can get information by having a conversation with a human-like assistant in the form of an avatar [2], a more or less "intelligent" being, a "simple digital assistant"; or by participating with "real intelligence with knowledge and emotions" as an autonomous virtual character [14] [8] . This system includes natural interaction elements such as tracking visitors' poses and positions, and the positions of physical objects (e.g. real books), to establish a context for the conversation. file://E:\mw2003\papers\sauer\sauer.html 5/27/2003 Figure 1: Digital Information Booth - Project "Info zum Anfassen" (tangible information). Based on that scenario, further innovative ideas and concepts are indicated by the IZA ("Info zum Anfassen", engl.: tangible information) project _[11 , which aims to develop a mixed reality platform for a booth scenario in cooperation with the visual computing department. Visitors are recognized by sensors and are saluted at the booth by avatars. Corresponding to the application scenario and conversation objectives, a talk is initiated. Here, instead of typical hypermedia, mouse- or menu-oriented interfaces, users can activate sensors or can physically select flyers or can show something, somewhere (gesture recognition) in order to select presentation modes or control the workflow of the story. The story engine is generic in the sense of different application scenarios (sales talk, infotainment, conversation, etc.), provides frameworks for various scenarios, and handles the device and media management. Grasbon and Braun describe a generic morphological approach to interactive storytelling in mixed reality environments [5] providing a non-linear interactive storytelling engine [3] based on a rule-based system implemented in JESS [4]. Figure 2 shows different multimodal interfaces integrated into the digital information booth of the IZA project. Exemplarily, microphones are used for speech recognition, a video camera is used for video recognition or a scanner for business cards is used to get in contact with the user and to get his email- address in order to provide personalized information. From the scientific-related point of view, there are several objects of research in Digital Storytelling which lead to new technology for multimedia applications: With new forms of scenario-based design, communication designers shall be included in an interdisciplinary design process for complex applications beyond screen design. Storytelling methods shall be involved to create drafts, design artefacts, and prototypes in a pre-production phase. To include storytelling aspects such as dramaturgy, character, and emotions in an interface, new forms of Conversational Interfaces are proposed. Further, the vision of writing as programming leads to new High-level APIs that allow application programming in a storytelling way. BEST COPY AVAILABLE 5 5/27/2003 file://E:\mw2003\papers\sauer\sauer.html \ I Figure 2: Digital Information Booth - multimodal interfaces A current technological trend is the development of intelligent assistance systems. In contrast to a tool that must support direct manipulation and should be handy, an assistant is expected to solve tasks autonomously, in parallel with other tools operated by the user. Proactivity is another criteria: in a museum, a trade show or other kiosk systems, a virtual being as a partner can facilitate orientation in artificial worlds by playing an active role in prompting user interaction. This interaction resembles interpersonal communication much more than interactions using traditional interactive tools. For this reason, currently anthropomorphous user-interface agents (avatars) within interactive (3D) worlds [11] are developed. Avatars are able to communicate by using human-like interaction techniques, such as lip-sync speech, facial expressions, and gestures. A conversational user interface system should be proactive, narrative, and emotional, and should represent a certain role metaphor. According to the role, not only is the avatar's behaviour and importance important, but also its location and timing. Application Examples Apart from digital information booths orinformation kiosks, there are other possible application areas using the same technology of user interaction design, interactive digital storytelling and multimodal user interfaces. For example, within the GEIST project a story engine based on the Propp model [12] is conceptualised as the control unit for dramaturgy [15][6], interactive workflow and further personalized and application-driven parameters in a mobile computer game. From the educational and learning point of view, the main goal of GEIST is to transmit historical information providing mixed reality technology. Users/pupils walk around at historical sites in Heidelberg and get information by virtual avatars presented on head-mounted displays. In the context of EMBASSI and MAP, users are provided with animated avatars as assistants integrated into entertainment electronics, e.g. presented on a TV screen. Examples for this exist in both multimedia home-entertainment and business applications. In order to improve acceptance of the user interface and to increase immersion, these applications are enhanced by human-like conversation rules for the interaction with user interface agents. 6 5/27/2003 fi le://E: \mw2003 \papers\ sauer\sauer.html Figure 3: TELEBUDDY - physical avatar visiting the EXPO exhibition 2000 in Hannover, Germany. TELEBUDDY represents a physical avatar equipped with different sensors respectively sense organs (see, listen, speak, show) enabling users to contact the Telebuddy via the internet and communicate via different channels for the various senses. For example, the Telebuddy can visit an exhibition and different users (user groups) can look through the eyes of the Telebuddy or speak to other visitors of the exhibition using the Telebuddy speech interface. With the art-E-fact project (art-E-fact: A Generic Platform for the Creation of Interactive Art Experience in Mixed Reality), an interdisciplinary team of computer scientists, media designers, artists, art historians, restorers and media publishers is reaching for a new stage of convergence of arts and technology. The project will create a platform to develop and explore new forms of creation, presentation and interaction with various kinds of artistic expression. Digital storytelling and Mixed Reality technologies enable new dimensions for artistic expression, and therefore will build the foundation for the new artistic applications. Therefore, the main objective of the art-E-fact project is to develop a generic platform for interactive storytelling in Mixed Reality to allow artists to create artistic expressions in an original way, within a cultural context between the virtual ("new") and the physical ("traditional") reality. The generic platform for artistic creation in mixed reality is based on a kernel that combines a virtual reality system with a scenario manager for interactive conversations with partially autonomous virtual characters. Further, components for media management and abstract device management enable flexibility and allow authors to design multimodal interactions on various abstraction levels. Results are positive: Providing interactive storytelling dialogue structures; instead of "navigation" metaphors in hypertext structures. Enabling the design of holistic spectator experiences by integration of design issues concerning content, story, characters, their modalities and used hardware; instead of being constrained to mere screen design with a fixed interaction metaphor. In summary, it is possible for artists to include anthropomorphic interactions such as speech, gestures, eye movement, and body poses into their design of mixed realities, and to direct lifelike avatars in order to act. 7 5/27/2003 file://E:\mw2003\papers\sauer\sauer.html Figure 4: art-E-fact - generic platform Art-E-fact is a generic platform for the creation of interactive art experiences in mixed reality. Sketched is an installation concept of visitors having conversations with virtual philosophers on the screen (left), with various options for physical props to interact (right). In the following sections of this paper, the authors describe new methods and concepts to adopt these basic technologies of interactive storytelling and user interaction design and to transmit them into a mobile scenario for museums as one representative of the wide range of edutainment applications. Here, game- oriented concepts are integrated to enhance the usability of developed concepts, respectively user acceptance, especially in the context of pupils and young visitors to museums. The difference between interactive storytelling and gaming [17] is that while both concepts try to attract attendees to walk through a story, within computer games users/players don't have any guarantee of solving the problem, which means it is possible to lose a game.You are always successful within a story. Another difference between story-telling and gaming is that digital story-telling always uses a dramatic concept . Braun,[7], Schneider [1] and Muller et al. [18] describe the usage of conversational user interfaces within these gaming vs. narrative storytelling environments. However currently there is a trend to merge both disciplines and to use storytelling metaphors within narrative environments in the context of cyberspace for computer games [9]. New methods and concepts The Transforming User Interface The concept of the Transforming User Interface, developed in 2001 by ion2s, tries to solve the problems of the dynamic changes of the mobile situation, context and environment around the user and his own priorities and needs. A digital device, the Mobile Companion, which is in use in four main areas (@home, @office, @car, stand-alone) interacts with the user across the visual, the auditive, and the haptical mode by sounds, images, speech, gestures etc. It transforms its user interface components when it is used in a new area or when the boundary conditions are changing. In the periphery the Mobile Companion and its User Interface components will be prepared for its next tasks. 8 5/27/2003 file://E: \mw2003 \papers \ sauer\ sauer.html ehome Figure 5: The 4 main areas (incl. periphery) for the Mobile Companion When it is used in one special field, for example at the office, the interaction principle is optimized for work at a desktop. Priority has the Graphical User Interface. When the Mobile Companion is integrated in the car, becoming a driving assistant system, the locus of the user's attention has to be on the driving process. It is difficult to communicate with the Mobile Companion by the Visual Mode. So the Graphical User Interface must be transformed to an optimized impulse-based interface and the auditive and gesture-based user interfaces become more important. For example, the result of a simple study shows that the changes in the driver's stress influence the Interface transformation. If stress is high, the auditive literfaces or gestures have priority, visual Interaction is very reduced and information concentration must be low. Higher density of content and the interaction per Graphical User Interface is possible in a low stress driving situation. Evaluation and studies of situations, environments, different user interface components and user contexts brought up a need for control equipment to govern the transformations . BEST COPY AVAILABLE 9 file://E:\mw2003\papers\sauer\sauer.html 5/27/2003 Figure 6: Sketches of example transformations for Smartphones and PDAs These principles and the main concept of the Transforming User Interface can be transferred to every Digital Appliance offering multimodal access and communication methods, and is used in variable and changing environments such as in exhibitions and museums. The present technology enhancements, for example tracking systems, offer the possibilities to understand the context and to focus the human being. This point is very important for building up experience environments for edutainment appliances. Focussing the user and getting him in conversation with Applications, Devices and other people enables User Interface Developers to generate innovative digital and interactive edutainment presentations, especially for kids. Kids Innovation Kids Innovation is a unit and a method of ion2s to develop user-centred interface concepts for the young target group. In cooperation with specialists for technical questions, storytelling, learning methods, education, pedagogy, etc. new concepts, applications and digital appliances are designed for kids in schools, in museums, in their private areas, etc. From the beginning the young user is completely involved as a full and most important member of the interdisciplinary project team. During the project the kids are organized in workshops for activities and evaluations to deliver useful information for the concepts, about the "needs" and "wants", and regarding demographical and statistical facts of their user group. The transfer of know-how and knowledge lets kids bring up new ways of thinking and is one synergetic effect. Kids Innovation projects help the clients and partners to evaluate their "user kids" during the development process and in the continuous usage of the digital application. Momuna A special edutainment concept, to integrate digital experience appliances in a museum, was based on Kids Innovation. The Momuna-concept (mobile museum navigator) is targeted for museums or theme parks, their educational departments and enrichment school programs. It combines simple mobile devices (Momuna.companion/.pad), standard workstations (Momuna.desk), tracking and location technologies (Momuna.dataloc) databases and servers (Momuna.node) and possibilities to integrate any kind of digital presentation devices and technologies (Momuna.addons). The Momuna-system enables institutions to get valuable feedback about their own exhibition concepts, the behavior of (young) visitors and how they deal with this new digital museum guide. The technologies and additional case studies help to support schools, classes or any other kind of visitor group within the three major phases of visiting a museum: preparation, execution and postprocessing. This sounds very technical but the concept focuses human-computer interaction principles, user interface modes, and users' groups and their needs. 1 0 file://E:\mw2003\papers\sauer\sauer.html 5/27/2003