PREFACE This book presents the latest advances ni the research of future computing and system design, as well as their relevant application, ni the wide field of human-computer interaction. The 381 papers presented in this volume were selected from those submitted to the Sixth International Conference on Human-Computer Interaction (HC! International '95) held ni Tokyo, 9-14 July 1995 with the support of a grant from the Commemorative Association for the Japan World Exposition (1970). A total of 1,298 individuals from 36 countries submitted their work for presentation at this first major international meeting on human-computer interaction held ni Asia. Among the submit- tals, only those judged to be of high quality were accepted for presentation. The papers accepted for verbal presentation, totaling 354, present recent advances in human interaction with comput- ers and related machines ni a variety of environments. The selected papers ni the areas of ergonomics, and social aspects of computer systems are included ni the accompanying Volume 2 entitled Symbiosis namuHr.o and Artifact: Human and Social Aspects of Human-Computer Interaction. We are greatful for the help of many organizations which made the congress successful, and would like to thank the following sponsors of the conference: Information Processing Society of Japan Institute for Electronics, Information and Communication Engineers Japan Ergonomics Research Society Public Health Research Center The Society for Instrument and Control Engineers and the following cooperating organizations: Architectural Institute of Japan Japan Association of Industrial Health Atomic Energy Society of Japan Japan Industrial Management Association Chinese Academy of Sciences Japan Institute of Office Automation Chinese Ergonomics Society Japan Management Association EEC-European Strategic Programme for Japan Society for Software Science and Research and Development in Information Technology Technology- ESPRIT Japan Society of Health Science Ergonomics Society of Taiwan Japanese Cognitive Science Society Finnish Institute of Occupational Health Japanese Society for Artificial Intelligence IEEE Systems, Man & Cybernetics Society Japanese Society for Science of Design IEEE Tokyo Section Korea Research Institute of Standards and Indian Society of Ergonomics Science Institute of Management Services (UK) National Institute for Occupational Safety & International Ergonomics Association Health (USA) National Institute for eht Improvement of ehT Illuminating Engineering Institute of Working Conditions dna Environment Japan )dnaliahT( ehT Institute of Electrical Engineers of Japan National Institute of Industrial Health (Japan) ehT Japan Society of Mechanical Engineers Society of Biomechanisms (Japan) ehT Japanese Association of Rehabilitation Software Psychology Society enicideM ehT Ergonomics Society of Korea ehT Society of Heating, Air Conditioning dna Sanitary Engineers of Japan. eW era most grateful ot eht following Board members rof their fine contributions ot eht organi- zation of eht conference: General Chair Advisory Committee Chair Yoshio Hayashi, Japan Kageyu Noro, Japan Vice Chair Organizing Committee Chair Hiroshi Tamura, Japan Takao Ohkubo, Japan Advisory Board Hideo Aiso, Japan Kazutaka Kogi, Japan Shun'ichi Amari, Japan Takao Shirasuna, Japan Takaya Endo, Japan Sadao Sugiyama, Japan laH Hendrick, U.S.A. Yotaro Suzuki, Japan Atsunobu Ichikawa, Japan ieK Takeuchi, Japan Kazumoto linuma, napaJ Thomas .J Triggs, Australia Hiroshi Kashiwagi, Japan Keiichi Tsukada, Japan Akinobu Kasami, Japan Masao Ueda, Japan Kakutaro Kitashiro, Japan Jtirgen .E Ziegler, Germany. eW thank, ni particular, eht Program Committee members who made valuable contributions to organizing eht program: Ame Aar/is, Norway Emiliano .A Francisco, The Philippines Munehira Akita, Japan Hiroshi Hamada, Japan Yuichiro Anzai, Japan )riahC( Hiroshi Harashima, Japan Kazuo Aoki, Japan Susan Harker, U.K. Albert .G Arnold, ehT Netherlands Martin Helander, Sweden Eiichi Bamba, Japan Herbert Heuer, Germany Nigel Bevan, U.K. Michitaka Hirose, Japan John .M Carroll, U.S.A. Erik Hollnagel, U.K. maY naS Chee, Singapore neK Horii, Japan Marvin .J Dainoff, U.S.A. Tohru Ifukube, Japan Miwako Doi, Japan Koichi Inoue, Japan Wolfgang Dzida, Germany Kitti Intaranont, Thailand Ray Eberts, U.S.A. Hiroo Iwata, Japan Klaus-Peter ,hcirnhi~F Germany Hiroyasu Kakuda, Japan iiv Katsuari Kamei, Japan Choon-Nam Ong, Singapore John Karat, U.S.A. Olov Ostberg, Sweden Osamu Katai, Japan Peter .G Poison, U.S.A. Takashi Kato, Japan Jens Rasmussen, Denmark Yosuke Kinoe, Japan Kazuo Saito, Japan Bengt Knave, Sweden Susumu Saito, Japan Richard .J Koubek, U.S.A Steven .L Sauter, U.S.A Masaharu Kumashiro, Japan Dominique .L Scapin, France Masaaki Kurosu, Japan Pentti Seppala, Finland Nahm Sik Lee, Korea Thomas .B Sheridan, U.S.A. Soon Yo Lee, Korea Ben Shneiderman, U.S.A. Xu Liancang, China Michael .J Smith, U.S.A. Holger Luczak, Germany T.F.M. Stewart, U.K. Thomas ,ilbui~L Switzerland Yasuo Sudoh, Japan Marilyn Mantei, Canada Yuzuru Tanaka, Japan Marvin Minsky, U.S.A. Yoh'ichi Tohkura, Japan Naomi Miyake, Japan Kim .J Vicente, Canada Hirohiko Mori, Japan Tomio Watanabe, Japan Masaki Nakagawa, Japan Runbai Wei, China Jakob Nielsen, U.S.A. Sakae Yamamoto, Japan Kazuhisa Niki, Japan Eiichiro Yamamoto, Japan Shogo Nishida, Japan Michiaki Yasumura, Japan Takeshi Nishimura, Japan Atsuya Yoshida, Japan Donald Norman, U.S.A Hidekazu Yoshikawa, Japan Katsuhiko Ogawa, Japan Richard Young, U.K. Takao Okubo, Japan This book, as well sa the conference program, could not have been completed without the out- standing effort of .sM Yoko Osaku, the secretariat for HCI International '95, and .rM Akira Takeuchi of the Musashi Institute of Technology. Yuichiro Anzai, Keio University Miwako Doi, Toshiba Corporation Hiroshi Hamada, NTT Hirohiko Moil, Musashi Institute of Technology Katsuhiko Ogawa, NTT Susumu Saito, National Institute of Industrial Health Symbiosis of Human dna Artifact Y. Anzai, K. Ogawa dna H. Mori (Editors) © 1995 Elsevier Science B.V. All rights reserved. Gesture Recognition for Manipulation in Artificial Realities Richard Watson *~ and Paul O'Neill b ~Computer Vision Group, Department of Computer Science, Trinity College, Dublin 2, Ireland bIona Technologies Ltd., 8-34 Percy Place, Dublin 4, Ireland In 1, we conclude that the flexible manipulation, by a human operator, of virtual objects in artificial realities is augmented by a gesture interface. Such an interface is described here and it can recognise static gestures, posture-based dynamic gestures, pose- based dynamic gestures, a "virtual control panel" involving posture and pose and simple pose-based trajectory analysis of postures. The interface is based on a novel, application independent technique for recognising gestures. Gestures are represented by what we term approzirnate splines, sequences of critical points (local minima and maxima) of the motion of degrees of freedom of the hand and wrist. This scheme allows more flexibility in matching a gesture performance spatially and temporally and reduces the computation required, compared with a full spline curve fitting approach. Training the gesture set is accomplished through the inter- active presentation of a small number of samples of each gesture. 1. THE GESTURE INTERFACE 1.1. Input and Output Streams The Gesture Interface receives two streams of input and produces one output stream: A stream of time-stamped homogeneous transformations describing the pose (po- sition and orientation) of the wrist with respect to the Control Space Base Frame. This input stream is generated by the GESTURE (POSE) subsystem. A stream of time-stamped values describing the posture of the hand and arm .2 Each value gives the magnitude of a particular degree of freedom of hand/arm posture. This input stream is generated by the GLAD-IN subsystem (i.e. the instrumented glove and exoskeleton) .3 *This research saw funded yb the Commission of the European Communities under the ESPRIT II krowemarF 2The pose and posture data yam be provided from yna source. During development of the Gesture Interface these input streams were produced from a level-hgih motion description simulation language .2 nI the later stages of development this simulation saw replaced yb input streams produced from pose/posture data recorded from the GLAD-IN ekil-evolG( Advanced Interface) and GESTURE (Wrist Pose calculation process) subsystems, and subsequently yb the live data. 3The angular magnitudes received from the GLAD-IN subsystem are assumed to correspond (within nevig tolerances) to the true angular magnitudes fo the hand/arm degrees fo freedom (dofs). nI other Each time the Gesture Interface recognises a physical gesture it sends at least a start and an end gesture notification to the client application. 2. GESTURE RECOGNITION In pattern recognition terms, the features extracted in this system, are critical points of a degree of freedom's motion or discontinuities. A discontinuity is a peak, a trough, or either the start or end of a plateau, as shown in figure .1 The classification stage is kaep ~ '/ kaep trough / start fo uaetalp ateau snoitavresbO trats morf emit .Ot • Ot Time Figure .1 Time-space pattern of a metacorpophlangeal joint (knuckle) in performing a gesture a template matching process where sequences of discontinuities for each degree of free- dom (dof) are compared against those extracted. A further classification stage calculates whether the gesture is acceptable according to several fit metrics. Analysing the input data from the proprioceptive glove and the pose calculation module, discontinuity extraction can be performed by analysing the angular velocity of a degree of freedom. Hand jitter is modelled simply by high frequency motion, thus the critical points are extracted using a low-pass filter. 2.1. Classification The interface module maintains a set of gesture templates, composed of sequences of discontinuities for sequences of degrees of freedom. The templates may be viewed as the axes of a multi-dimensional gesture space; thus the aim of the classifier is to firstly calculate the axis to which a given set of observed motion discontinuities is closest, and then to decide whether this is close enough given a set of distance metrics. The first process of mapping a set of observed discontinuities to a gesture subspace i.e., matching sequences of discontinuities, can be formulated as a finite state acceptor (FSA), shown here as the 5-tuple, M~ =< Q,I, ,5 ,0q F >. M~ accepts an instance of the correct discontinuity pattern, for a degree of freedom, j and a gesture class, ,c where the state set, Q, is the set of partial pattern matches, the input alphabet, I, is the set of ,sdrow the GLAD-IN calibration procedure si assumed to eb evitceffe enough ot reduce/remove the need rof cificeps-resu training fo the Gesture .ecafretnI discontinuity types, the transition function, ,5 is determined by the temporal sequence of discontinuities trained for this template, the initial state, 0q is the first discontinuity in the sequence, and F _C Q, acceptable halting states, is the final discontinuity. An example discontinuity pattern and its representation in this formulation is shown in figure .2 Dof i st plateau end plateau e-----O Dofi max .e st plateau • min .... \, e---- Time min st plateau end plateau e min in ( ) ( / i ! (( ...... /) Time Figure .2 Template Discontinuity pattern for Figure .3 Template pattern with a recur- a single degree of freedom and a labelled di- ring discontinuity. graph corresponding to its FSA. The matching process is made more complex by the small number of discontinuity types. Consider, for example, the problem occuring where a template with a recurring discontinuity, as in figure .3 The first two discontinuities have been matched. As a minimum is observed, it is not clear whether this is the first or the third discontinuity. Thus, a new matching attempt must be started as another instantiation of the FSA for this degree of freedom to cover the former case. 2.2. Feature Computation Most gesture templates have a small number of discontinuities, thus the set of gestures which can be unambiguously represented is correspondingly small. For example, the set of static gestures, consisting of a start plateau followed by an end plateau for each degree of freedom are represented identically. A set of features and corresponding metrics further characterise and disambiguate ges- tures, at 2 levels of detail: per discontinuity, i and per degree of freedom (sequence of discontinuities), j. These features are described formally, where C represents the set of gesture classes or templates. Thus qi~j(x) is the observed magnitude of the discontinuity i, degree of freedom j, in gesture template c. Q[j(x) is the equivalent discontinuity in template .c There are also interest conditionals: ;(c, j), which is true when the degree of freedom j is significant for classification of gesture class c and ~(7, ,c j), which is true when the metric f is significant for the degree of freedom j, and gesture class .c For each metric, gesture class, degree of freedom and discontinuity used there is a corresponding acceptability threshold, ,e computed by the Gesture Training Module. Discontinuity i level metrics Absolute Magnitudes Q(j,x) " Vi Iqi,j(x) C - Qi,j(x)l C < ,le c~Q ~ (x) A ~(Q(j,x), c, j) Absolute Timestamps Q(j, t)'Vi Iqi~,j(t)- )t(j,~iQ < e~:Q(t) A ~(Q(j, t), c, j) Degree of Freedom j level metrics Aggregate discontinuity level metrics l-I(/): VjQ(j,x) A Q(j, t) A ~(c, j) Range of Motion A(x) • jV 5(x) - 5~(x) < e.,j A,c (x) A ~(A(x), c, j) A ~(c,j) where 5(x) = I max(q:j(x)) - min(q:j(x)) and 5~(x) - I max(Qj(x)) - min(Q.~j(x))l Spatial Scaling Uniformity S(x)'Vj(N(j,x) < e'jS(x)A ~o(S(x),c,j))A ;(c, j) ,Ciq ,ciQ)X(j j(x) I where N(j, x) = Ei--1 c X 2 Temporal Scaling Uniformity $(t) " Vj(N(j,t) < )t(S),~.e A ~($(t),c,j)) A ;(c,j) Ei=I I qi~,j(t)Q~,j(t) where N(j, t) = V/~ill qi,j(t) c 2 ~i:1 I ,,w.cQ /2)t(" Hence the gesture class c is matched if : H(i) A A(x) A S(x) A S(t) Figures 4 and 5 show the scope for spatial and temporal scaling in this approach. For one degree of freedom, a set of observed discontinuities is matched to corresponding template discontinuities. Disregarding the absolute values of degree of freedom magnitude and timestamp cannot strictly be called scale-invariance, since ignoring these values allows many types of pattern warping. The subset of these metrics to employ for a particular gesture is specified by the user in an interactive training procedure. 2.3. Wrist Pose The pose of the wrist is provided by GESTURE as a homogeneous transformation from which three degrees of freedom for position and three for orientation may be extracted. Dynamic gestures involve movement, and hence naturally the position and orientation of the wrist. The pose of the wrist may be important in one of several ways: - Translating static gestures (holding hand posture constant and changing hand pose) to add emphasis or parameters to the original meaning, or to easily multiply the number of gestures recognised by differentiating the direction of translation as in Fels' system .3 - In a gesture, for example, where the posture is a point, the direction along which this point is made may be important, or it may be necessary to actually translate the posture in the desired direction. Observed Pattern Template Observed Pattern Template @ ~!!iiiiii-- / o ¢ Time Time Figure .4 Spatial scaling Figure .5 Temporal scaling Patterns traced out by the position of a fingertip are examples of gestures, a circle, meaning rotate, or an X drawn over an object to mean remove it from view. Positional trace pattern gestures are handled within the framework provided by the classifier by treating discontinuities in (z, y, z) position identically to posture discontinu- ities. Thus, circles and X patterns, for example, have templates consisting of patterns of temporally ordered discontinuities in the x, y and z axes. To prevent spurious matches it is necessary to apply fit metrics to the circle trace gesture: minimum diameter and diameter ratio are employed. 3. GESTURE TRAINING MODULE The purpose of the Gesture Training Module is to semi-automatically compute a rep- resentation for each physical gesture. The required representation will vary from user to user. Usually this variation will lie only in discontinuity magnitudes and time-stamps. The purpose of asking the user to perform multiple samples of each gesture is to obtain an idea of the natural variation in the way the person makes the gesture. There are two principal points to note about the gesture training mechanism described in this section. It is only necessary to present a small number of samples of each physical gesture to the system. Empirical tests show that five samples of each physical gesture are sufficient. Also, the end-product of training is an ezplicit, understandable representation of each gesture. The information required to fully describe a physical gesture may be broken into two categories: (i) Automatically Generated. This information is computed from pre- sented gesture samples. It consists :fo discontinuity patterns, discontinuity magnitudes and time-stamps, acceptance tolerances for metrics using the magnitudes and time-stamps and jitter tolerance (used as a threshold during discontinuity extraction). Consider a sin- gle degree of freedom. If the discontinuity patterns based upon each of the samples are not identical to each other then a majority voting algorithm is invoked. (ii) User-Supplied. This consists of decisions about the appropriateness of metrics to apply to gestures and de- grees of freedom. During training the user typically "refreshes" an existing set of physical gesture templates, through recomputation of the (user-specific) automatic information. In this case it is not necessary for the user to supply information about applicable metrics. 01 .4 CONCLUSIONS 4.1. Results An arbitrary number 4 of static gestures can be recognised from the Irish single-handed deaf alphabet as can posture-based dynamic gestures such as "Come Here ''5 and "Thumb Click ''6. The following pose-based dynamic gestures can be recognised based upon their discontinuity patterns: "Circle" 7 and "X"S. By employing these gestures, artificial reality commands such as navigation, point and click ("mouse emulation"), view point manipulation (zooming, panning etc.), meta- commands (such as resetting the viewpoint or quitting from the system), and manipu- lation of graphical objects (i.e., grasping, their creation and deletion) can be effected. These virtual world commands are documented in more detail in a further paper 4. 4.2. Future Work Future work will concentrate on development of a more flexible discontinuity pattern representation which allows variability to be expressed elegantly and orientation-invariant descriptions of pose-based gestures. At present the computational task of recognising gestures is O(n), where n is the number of gesture classes (or templates). A method of constructing a tree (or hash table) of partial discontinuity sequence matches would (in theory) reduce this complexity to O(log n). REFERENCES .1 Richard Watson. A Survey of Gesture Recognition Techniques. Technical Report TCD-CS-93-11, Department of Computer Science, Trinity College Dublin, July 1993. Available at ftp://ftp.cs.tcd.ie/pub/tcd/tech-reports/reports.93/TCD-CS-93-11.ps.Z. 2. Richard Watson. A Gesture Simulation Language. Technical Report TCD-CS-93-12, Department of Computer Science, Trinity College Dublin, July 1993. Available at ftp://ftp, cs. t cd.ie / pub / tcd / t ech-rep orts / reports. 93 / T C D- C S- 93-12. ps.Z. 3. S. Sidney Fels and Geoffrey E. Hinton. Building adaptive interfaces with neural networks: The glove-talk pilot study. In Human-Computer Interaction--INTERACT ,09' pages 683-688. IFIP, Elsevier Science Publishers B.V. (North-Holland), 1990. 4. Richard Watson and Paul O'Neill. A Flexible Gesture Interface. In Wayne Davis, editor, Proceedings of Graphics Interface '95, Montreal, Canada, May 1995. 4The correct recognition of gestures based upon small differences ni thumb position has proved difficult (largely due to calibration difficulties); and it si not physically possible to make some gestures while wearing the glove, due to physical interference between the sensors. nA example of this type of gesture si where one finger must eil flat upon another. 5The initial posture of this gesture si a flat-hand. The forefinger si flexed and then extended again ni one smooth motion. 6Thumb flexion and way si brought from its minimum value to its maximum value and then back to its minimum value ni one smooth motion, while the other degrees of freedom maintain a static point gesture. 7The user traces a circle ni space, with sih wrist. The circle gesture has been problematic ni that it si difficult for the user to make a precise (or even approximate) circle. In addition, the discontinuity pattern observed during a circular motion ni D3 space depends upon the orientation of the circle and the direction ni which its boundary si traced. SThe user traces an X pattern ni space with sih wrist. sisoibmyS of namuH dna tcafitrA .Y ,iaznA .K awagO dna .H Mori )srotidE( 11 © 5991 reiveslE ecneicS B.V. llA sthgir .devreser Hand Gesture Recognition Using Computer Vision Based on Model-matching Method Nobutaka Shimada, Yoshiaki Shirai and Yoshinori Kuno Dept. of Mechanical Engineering for Computer Controlled Machinery, Osaka University, Yamadaoka 2-1, Suita, Osaka, 565 Japan This paper proposes a method of 3-D model-based hand pose recognition from monoc- ular silhouette image sequences. The principle of the method is to search for the hand pose which matches best to a silhouette in an image among possible candidates generated from the 3-D hand model. The number of candidates is reduced by considering the locations of features extracted from the silhouette, the prior probability of shape appearance, and the sensitivity of the shape change to the model parameter change. In addition, the multiple solutions are preserved to obtain the globally optimal solution over a long sequence. 1. INTRODUCTION There has been a strong demand for automatic hand gesture recognition for human interfaces. Methods for hand gesture recognition are classified into two categories: use of special gloves with sensors and use of computer vision techniques. Although the former can give reliable information, the connection cables limit the human movement. Therefore, interests in the latter have been increasing. Several researchers including J.Davis, et al.1 have proposed hand gesture recognition systems in which marks are attached on finger tips, joints, and wrist. Although these methods are suitable for real-time processing, it is not convenient for users to wear such marks. Another approach, proposed by M.Mochimaru, et al.2, tries to match the image generated from a 3-D hand shape model and the silhouette image. The overlapping area alone is insufficient to evaluate the similarity of the shape of the silhouette. J.Rehg3, et al. have proposed a method to use the constraints of global shape features in order to solve the kinematic equations of the hand model. These methods 2,3 assume that each part of the object moves a little during an image sampling interval. However, this assumption does not hold when the finger motion is quick as in usual hand gestures. There are even such cases in actual hand gestures that one or more fingers suddenly vanish in the silhouette because of occlusion by the other fingers or the palm. We propose a method of hand pose estimation using 2-D shape features extracted from the silhouette. It can robustly estimate hand poses without any marks. We actively generate pose candidates from the 3-D model, and search for the best-matched pose to the silhouette, using not only the overlapping areas as Kameda4, but characteristic shapes in the silhouette. The matching degree of each candidate si expressed as the probability to integrate the different sort of the degree of matching. The search space is large due to large degrees of freedom of a human hand. To reduce the search space, we propose three frameworks. The first one is the use of structural