VISUAL LANGUAGES MANAGEMENT AND INFORMATION SYSTEMS EDITORIAL BOARD: A. Bensoussan, Le Chesnay, France Y. Oishi, Tokyo, Japan S. K. Chang, Pittsburgh, Pennsylvania N. Miiller, Osnabriick, Federal H. W. Gottinger, Bielefeld, Federal Republic of Germany Republic of Germany G. Pask, Richmond, England M. D. Intriligator, Los Angeles, D. W. Peterson, Durham, California North Carolina R. E. Kalman, Zurich, Switzerland M. Thoma, Hannover, Federal R. Kling, Irvine, California Republic of Germany A. Klinger, Los Angeles, California P. P. Wang, Durham, North Carolina R. W. Liu, South Bend, Indiana A. B. Whinston, West Lafayette, T. Marschak, Berkeley, California Indiana B. H. McCormick, College Station, L. A. Zadeh, Berkeley: California Texas R. Michalski, Urbana-Champaign, Illinois LANGUAGES FOR AUTOMATION Edited by Shi-Kuo Chang MANAGEMENT AND OFFICE INFORMATION SYSTEMS Edited by Shi-Kuo Chang VISUAL LANGUAGES Edited by Shi-Kuo Chang, Tadao Ichikawa, and Panos A. Ligomenides A Continuation Order Plan is available for this series. A continuation order will bring delivery of each new volume immediately upon publication. Volumes are billed only upon actual shipment. For further information please contact the publisher. VISUAL LANGUAGES EDITED BY SHI-KUO CHANG University of Pittsburgh Pittsburgh, Pennsylvania TADAO ICHIKAWA Hiroshima University Higashi-Hiroshima, Japan AND PANOS A. LIGOMENIDES University of Maryland College Park, Maryland PLENUM PRESS • NEW YORK AND LONDON Library of Congress Cataloging in Publication Data Visual languages. (Management and information systems) Includes bibliographical references and index. 1. Image processing-Digital techniques. 2. Programming languages (Electronic computers) I. Chang, S. K. (Shi Kuo), 1944- . II. Series. TA1632.V56 1986 006.6'6 86-25357 ISBN-13: 978-1-4612-9010-0 e-ISBN-13: 978-1-4613-1805-7 DOl: 10.1007/978-1-4613-1805-7 First Printing-December 1986 Second Printing-July 1989 © 1986 Plenum Press, New York Softcover reprint of the hardcover 1s t edition 1986 A Division of Plenum Publishing Corporation 233 Spring Street, New York, N.Y. 10013 All rights reserved No part of this book may be reproduced, stored in a retrieval system, or transm~tted in any form or by any means, electronic, mechanical, photocopying, microfilming, recording, or otherwise, without written permission from the Publisher CONTRIBUTORS K. ASS MANN • Institut fur Mathematik und Datenverarbeitung in der Medizin, Universitiitskran kenhaus Eppendorf, Hamburg, Federal Republic of Germany S. K. CHANG. Department of Computer Science, University of Pittsburgh, Pittsburgh, Pennsylvania KAI CHU • Department of Computer Information Systems, Humboldt State University, Arcata, California OLIVIER CLARISSE. AT&T Bell Laboratories, Software Automation Group, Naperoille, Illinois MASAHITO HIRAKAWA. Faculty of Engineering, Hiroshima University, Higashi-Hiroshima, japan K. H. HOHNE • Institut for Mathematik und Datenverarbeitung in der Medizin, Universi- tiitskrankenhaus Eppendorf, Hamburg, Federal Republic of Germany TADAO ICHIKAWA. Faculty ofE ngineering, Hiroshima University, Higashi-Hiroshima,japan ROBERT J. K. JACOB • Naval Research Laboratory, Washington, D.C. ERLAND jUNGERT • National Defense Research Institute, Applied Electronics Department, Lin- kOping, Sweden TOHRU KIKUNO • Faculty of Engineering, Hiroshima University, Higashi-Hiroshima, japan MARGARET A. KORFHAGE • Trammell Crow Company, Dallas, Texas ROBERT R. KORFHAGE • Department of Computer Science, Southern Methodist University, Dallas, Texas STEPHEN M. KOSSL YN • Department of Psychology and Social Relations, Haroard University, Cambridge, Massachusetts FRED LAKIN • Veterans Administration Medical Center, Palo Alto, California JAMES A. LARSON. Honeywell Corporate Systems Development Division, Bloomington, Minnesota P. A. LIGOMENIDES • Electrical Engineering Department, University of Maryland, College Park, Maryland jUN'ICHI MIYAO • Faculty of Engineering, Hiroshima University, Higashi-Hiroshima, japan NORIAKI MONDEN • Faculty ofE ngineering, Hiroshima University, Higashi-Hiroshima,japan GABRIELE ROHR • IBM Science Center, Heidelberg, Federal Republic of Germany MARCIA S. RUNYAN. Department of Computer Science, University of Washington, Seattle, Washington NAN C. SHU • IBM Academic Information Systems, Los Angeles Scientific Center, Los Angeles, California KAZUO SUGIHARA • Faculty of Engineering, Hiroshima University, Higashi-Hiroshima, japan MASA YUKI TAKAYAMA • Faculty of Engineering, Hiroshima University, Higashi-Hiroshima, japan v VI CONTRIBUTORS MINORU TANAKA. Faculty of Engineering, Hiroshima University, Higashi-Hiroshima, Japan STEVEN L. TANIMOTO • Department of Computer Science, University of Washington, Seattle, Washington MICHAEL J. TAUBER. IBM Science Center, Heidelberg, Federal Republic of Germany R. VENEMA • lnstitut for Mathematik und Datenverarbeitung in der Medizin, Universitiitskran kenhaus, Eppendorf, Hamburg, Federal Republic of Germany ROBERT J. WEBER. Department of Psychology, Oklahoma State University, Stillwater, Oklahoma NORIYOSHI YOSHIDA. Faculty of Engineering, Hiroshima University, Higashi-Hiroshima, Japan IWAO YOSHIMOTO • Faculty of Engineering, Hiroshima University, Higashi-Hiroshima, Japan PREFACE This book is intended as both an introduction to the state-of-the-art in visual languages, as well as an exposition of the frontiers of research in advanced visual languages. It is for computer scientists, computer engi neers, information scientists, application programmers, and technical managers responsible for software development projects who are inter ested in the methodology and manifold applications of visual languages and visual programming. The contents of this book are drawn from invited papers, as well as selected papers from two workshops: the 1985 IEEE Workshop on Lan guages for Automation-Cognitive Aspects in Information Processing, which was held in Mallorca, Spain, June 28-30, 1985; and the 1984 IEEE Workshop on Visual Languages, which was held in Hiroshima, Japan, December 7-9, 1984. Panos Ligomenides and I organized the technical program of LFA '85, and Tadao Ichikawa and I organized the techni cal program of VL '84. Both workshops have now become successful annual events in their own right. The intersecting area of visual languages and visual programming especially has become a fascinating new research area. It is hoped that this book will focus the reader's attention on some of the interesting research issues as well as the potential for future applications. After reading this book, the reader will undoubtedly get an impression that visual languages and the concept of generalized icons can be studied fruitfully from many different perspectives, including computer graphics, formal language theory, educational methodology, cognitive psychology and visual design. Cooperative and interdisciplinary research can lead to a better understanding of the visual communication process for developing an effective methodology to design the next generation of visual languages. Shi-Kuo Chang University of Pittsburgh vii CONTENTS INTRODUCTION VISUAL LANGUAGES AND ICONIC LANGUAGES Shi-Kuo Chang 1 PART I VISUAL PROGRAMMING LANGUAGES 1. VISUAL PROGRAMMING LANGUAGES: A PERSPECTIVE AND A DIMENSIONAL ANALYSIS Nan C. Shu 11 2. SPATIAL PARSING FOR VISUAL LANGUAGES Fred Lakin 35 3. A VISUAL PROGRAMMING ENVIRONMENT FOR DESIGNING USER INTERFACES Robert J. K. jacob 87 4. A VISUAL LANGUAGE FOR FORMS DEFINITION AND MANIPULATION Kazuo Sugihara,jun'ichi Miyao, Masayuki Takayama, Tohru Kikuno, and Noriyoshi Yoshida 109 5. VISUAL LANGUAGES FOR DATABASE USERS james A. Larson 127 PART II ICONIC AND VISUAL INFORMATION PROCESSING LANGUAGES 6. VICON: A VISUAL ICON MANAGER Olivier Clarisse and Shi-Kuo Chang 151 7. PLAY: AN ICONIC PROGRAMMING SYSTEM FOR CHILDREN Steven L. Tanimoto and Marcia S. Runyan 191 8. CRITERIA FOR ICONIC LANGUAGES Robert R. Korfhage and Margaret A. Korfhage 207 9. HI-VISUAL: A LANGUAGE SUPPORTING VISUAL INTERACTION IN PROGRAMMING Masahito Hirakawa, Noriaki Monden, [wao Yoshimoto, Minoru Tanaka, and Tadao Ichikawa 233 10. THE ISQL LANGUAGE: A SOFTWARE TOOL FOR THE DEVELOPMENT OF PICTORIAL INFORMATION SYSTEMS IN MEDICINE K. Assman, R. Venema, and K. H. Hohne 261 IX x CONTENTS 11. A CONCEPTUAL IMAGE DATA MODEL FOR IMAGE INFORMATION RETRIEVAL Erland Jungert 285 PART III COGNITIVE ASPECTS IN VISUAL INFORMATION PROCESSING 12. COMPUTER GRAPHICS AND MENTAL IMAGERY Robert J. Weber and Stephen M. Kosslyn 305 13. USING VISUAL CONCEPTS Gabriele Rohr 325 14. THE COGNITIVE ASPECTS OF CHINESE CHARACTER PROCESSING Kai Chu 349 15. TOP-DOWN DESIGN OF HUMAN-COMPUTER INTERFACES Michael J. Tauber 393 16. THE EXPERIENTIAL KNOWLEDGE BASE AS A COGNITIVE PROSTHESIS Panos A. Ligomenides 431 INDEX 453 INTRODUCTION: VISUAL LANGUAGES AND ICONIC LANGUAGES SHI-Kuo CHANG The term "visual language" means different things to different people. To some, it means the objects handled by the language are visual. To others, it means the language itself is visual. To the first group of people, "visual language" means "language for processing visual information," or "visual information processing language." To the second group of people, "vis uallanguage" means "language for programming with visual expressions," or "visual programming language." In visual information processing languages, the objects to be dealt with usually have an inherent visual representation. They are images or pictorial objects which are associated with certain logical interpretations. On the other hand, the languages themselves may not have a visual rep resentation. These languages are usually based upon traditional "linear" languages, enhanced by library subroutines or software packages to deal with visual objects. Application domains of visual information processing languages include image processing, computer vision, robotics, image database management, office automation, image communications, etc. In visual programming languages, the objects to be dealt with usually do not have an inherent visual representation. They include traditional data types such as arrays, stacks, queues, and application-oriented data types such as forms, documents, databases, etc. To achieve a user-friendly man-machine interface, we would like to present these objects visually. For the same reason, the languages themselves also should be presented visually. In other words, programming constructs and how to combine these programming constructs are visually presented. Application domains SHI-KUO CHANG • Department of Computer Science, University of Pittsburgh, Pitts burgh, Pennsylvania 15260.