Research Reports ESPRIT Project 5345 . DIMUS . Volume 1 Edited in cooperation with the Commission of the European Communities S. Pfleger J. Goncalves D. Vernon (Eds.) Data Fusion Applications Workshop Proceedings Brussels, November 25, 1992 Springer-Verlag Berlin Heidelberg New York London Paris Tokyo Hong Kong Barcelona Budapest Volume Editors S. Pfleger Technical University of Munich Orleansstr. 34, D-81667 Munich, Germany J. Gonc;:alves Commission of the European Communities Joint Research Center, 1-21020 Ispra, Italy D. Vernon Commission of the European Communities Rue de la Loi 200, B-1049 Brussels, Belgium ESPRITProject 5345 "Data Integration in MultisensorSystems (DIMUS)" belongs to the Subprogramme "Information Processing Systems and Software" of ESPRIT, the European Strategic Programme for Research und Development in Information Technology supported by the Comission of the European Communi ties. The project investigates the advanced technology of multisensoring in order to obtain correct and complete scene information in monitoring and control appli cations with safety-critical constraints. The objective is to design and develop a system supporting a human operator in the interpretation of situations occurring in a metro (subway) station enviroment. A prototype system is being developed containing a set of tools for the integration and fusion of visual and non-visual data provided by different kind of sensors. CR Subject Classification (1991): H.4, 1.0, K.6 ISBN-13: 978-3-540-56973-2 e-ISBN-13: 978-3-642-84990-9 DOl: 10.1007/978-3-642-84990-9 Publication No. EUR 15247 EN of the Commission of the European Communities, Dissemination of Scientific and Technical Knowhow Unit, Directorate General Information Technologies and Industries, and Telecommunications, Luxembourg LEGAL NOTICE Neither the Commission of the European Communities nor any person acting on behalf of the Commission is responsible for the use which might be made of the following information. © ECSC - EEC - EAEC, Brussels - Luxembourg, 1993 Typesetting: Camera-ready by authors 45/3140 -543210 - Printed on acid-free paper Foreword The 1992 ESPRIT Conference and Exhibition which was held in Brussels during the week of November 23rd to 27th was remarkable for the number of technical workshops which it hosted. One of these workshops was devoted to the topic of "Data Fusion Applications" and its success is indicative of the healthy rapport which now exists between the industrial practitioners on the one hand and the research community on the other. ESPRIT - the European Strategic Programme for Research and Development in Information Technology -has played no small part in fostering this relationship. Data Fusion itself is, and will remain, an important topic in information technology. The reason is straightforward: as we attempt to expand the application domains of information technology to encompass ever more sophisticated systems and environments, the data which is derived from a single source cannot be assumed to be adequate to characterize fully the application domain. The ability to combine data derived from several sources, data fusion, to provide a coherent, informative and useful characterization then assumes a key role in the development of robust IT systems. This volume brings together a valuable collection of experiences in building systems which depend on successful data fusion and it constitutes a useful point of reference in the development of advanced IT systems in ESPRIT. June 1993 D. E. Talbot Preface One is faced with many problems when designing computer-based systems which have to interpret information as it arises in natural environments. Chief among these problems is the apparent inadequacy of the available data which characterizes the application at hand, and often we can only form a partial picture of what the data is supposed to convey. On the other hand, we have the self-evident ability of humans to work extremely well in equally testing circumstances. A striking example of this ability concerns the detection of Coal Workers Pneumoconiosis (CWP), an occupational disease affecting coal miners. The early detection of CWP is effected by identifying small round opacities in a sequence of chest radiographs which have been generated at regular intervals over time. In spite of the absence of an independent reference model, CWP readers achieve remarkable performance and consistency in the classification of radiographs into one of twelve categories in the system approved by the International Labour Office, exhibiting an inter- and intra-reader standard variation which is better than one category. The surprising factor is that CWP readers substantially disagree in pin pointing the cues, i.e. small round opacities, which form the basis for the final classification. In other words, there is strong agreement in the overall classification in spite of the fact that the same radiograph manifests different features to different readers. It can be argued that the difference in the identification of these features is a consequence of different learning sets and procedures. Whatever the case, it appears that the successful interpretation depends at least as much on the way the reader collates - or fuses - the features as much as it does upon the individual features -the data - themselves. So how do we deal with such problems in computer-based systems? There are two obvious, and popular, solutions. The first is to improve significantly the characteristics of the sensors which are responsible for gathering the data in the first place, in the belief that with better, more reliable, and less noisy data, the problem will solve itself. While this is a valid, and frequently a necessary approach, it is by no means a panacea for these types of applications. The second solution is to try to glean as much data as possible from as many sources as possible and then to 'combine' this data in order to reconstruct and characterize the complete application domain. Unfortunately, this data fusion is not at all a trivial task and there is no single unified and proven solution which is applicable in all circumstances. Nonetheless, there are many plausible and useful approaches which can be used, and are being used, to solve particular applications in data fusion. At the 1992 ESPRIT Conference and Exhibition, the DIMUS project organized a workshop on Data Fusion Applications which emphasized the solutions to practical problems. The papers in this book describe the projects and the work presented at this workshop. A number of other papers were included later, in view of their relevance. VIII As one reads through this volume, one cannot but be struck by the extraordinarily wide variety of application domains in which data fusion plays a role: from the increasingly-important medical area, through industrial control, to surveillance applications in the nuclear and the transport industries. Without doubt, the scientific aspects of data fusion, i.e. the formal modelling of different information sources and their commonality which underpins successful fusion, is maturing. Equally, the validation of these theories and techniques through application has an important role to play in this maturation process. Together, they will see the development of more robust data fusion capabilities and, con sequently, we will increasingly find complex problems in poorly-characterized domains yielding to solution, to the benefit of industry and the Research and Development community alike. This book represents a unique snap-shot of the current state of play in the development and validation of technique for data fusion in a wide variety of applications, ranging from the fusion of visual and non-visual data to the integration of multi-sensor systems, and embracing many of the essential issues of, e.g., architectures, real-time responses, and safety-critical requirements. It forms a valuable reference point against which we can measure our progress in data fusion research and development, and it represents a useful foundation upon which we can build. s. June 1993 Pfleger J. Gon~alves D. Vernon Contents Combining Two Imaging Modalities for Neuroradiological Diagnosis: 3D Representation of Cerebral Blood Vessels .... ........................ 1 M. Bahner, J. Dick, B. Kardatzki, H. Ruder, M. Schmidt, A. Steitz, C. Bertram, D. Hentschel, T. Hildebrand, E. Hundt, R. Kutka, S. Stier, G. Gerig, T. Koller, O. Kiibler, G. Szekely Hybrid Inference Components for Monitoring of Artificial Respiration ................................................................................. 17 K. Giirtner, S. Fuchs, H. Jauch Information Fusion in Monitoring Applications using the Category Model ............................................................................................. 27 W. Steinke A Flexible Real-Time System for Vessel Traffic Control and Monitoring ..... .................. .................. .................. .......................... ......... 38 L. Stefanelli ESPRIT Project AZZURRO: Data Fusion for Marine Protection ........... .... .................... .................... ................ .... ........ .... ... 44 S. Ghelfo ESPRIT Project DIMUS: Data Integration in Multisensor Systems ............. 50 F. Benvenuto, M. Ferrettino, M. Pasquali, F. Perotti, P. Verrecchia A Reflex-Based Approach to Fusion of Visual Data ..................................... 61 A. Bozzoli, M. Rossi, R. BarbO, B. Caprile, G. Carlevaro Sensor Fusion in a Peg-In-Hole Operation with a Fuzzy Control Approach ............... .............. ........ ...... ................ ................ .... 71 J. Zhang, J. Raczkowsky Fuzzy Logic Techniques for Sensor Fusion in Real-Time Expert Systems ............................................................................ 79 JA Aguilar-Crespo, J. M. Dominguez, E. de Pablo, X. Alanuin Task-Directed Sensor-Planning 87 G. Grunwald, G.D. Hager x Incremental Map-making in Indoor Environments ................... ................ ... 102 A. Ekman, D. Stromberg Image Segmentation Improvement with a 3-D Microwave Radar ............... 115 A. Siebert, J. Ostertag, B. Radig, M. Rozman, J. Detlefsen, J. Bernasch A Vectorial Definition of Conceptual Distance for Prototype Acquisition and Refinement ......................................................................... 123 C. Moneta, G. Vernazza, R. Zunino Distributed Knowledge-based Systems for Integration of Image Processing Modules ...................................................................... 133 C. Regazzoni Spatial Fusion of Multisensor Visual Information for Crowding Evaluation .................................................................................... 155 M. Peri, C. Regazzoni, A. Tesei, G. Vernazza Robust Multisensor Fusion in Underground Stations ................................... 172 S. Pfleger, A. Milano On Tracking Edges 183 M. Tistarelli Force and Position Sensing Resistors: An Emerging Technology............... 196 J. Hagen, M. Witte Fusing Two Views using Object Motion .... ........................ ...... .................... 204 D. Hogg, A. Baumberg Mobile Robotics for the Surveillance of Fissile Materials Storage Areas: Sensors and Data Fusion ...................................................... 214 J. G. M. Gom;alves, G. Campos, V. Santos V. Sequeira, F. Silva Data Fusion for Environmental Monitoring ...... .......... .... ........................ ..... 246 T. Wittig, H.-N. Pham Concluding Remarks: Advanced European Research on Data Fusion ... .................. .... .......... ............ .... ............ .... ...................... ..... 262 K.c. Varghese, S. Pfleger Contributors .... .................. .................. ...................... ...................... .... .......... 266 Combining Two Imaging Modalities for Neuroradiological Diagnosis: 3D Representation of Cerebral Blood Vessels Michael Bahner, Jiirgen Dick, Bernd Kardatzki, Hanns Ruder, Matthias Schmidt, Amo Steitz Theoretical Astrophysics, University of TUbingen Carsten Bertram, Dietmar Hentschel, Thomas Hildebrand Siemens Medical Systems, Erlangen Eckart Hundt, Robert Kutka, Sebastian Stier Siemens Corporate Research, MUnchen Guido Gerig, Thomas Koller, Olaf KUbler, Gabor Szekely Communication Technology Laboratory, Image Science Division, ETH ZUrich Abstract Today the integration of information from different imaging modalities in medicine such as Computer Tomography or Magnetic Resonance Imaging (MRI) is left to the physician and gets little support from computers. In the case of neuroradiological diagnosis, information about cerebral blood vessels is available from 3D volume data from Magnetic Resonance Angiography (MRA) and from 2D images generated by Digital Subtraction Angiography (DSA). The DSA images have a higher resolution than MRA data, and therefore neuroradi ologists are highly interested in a 3D reconstruction of cerebral blood vessels from different DSA projections. On the other hand, MRA contains important functional information, the velocity of blood flow. This paper describes work in progress to make available to the physician the full 3D information from both imaging modalities including an approach to 3D reconstruction from DSA im ages which makes use of the MRA data. The 3D DSA reconstruction also opens the way to an integration of information from DSA with completely different types of information, for example information on anatomical structure or soft tissue from MRI. An integral part of this work is a pilot system for clinical vali dation. As a typical case the neuroradiologist is interested in a 3D representation of the blood vessels surrounding an aneurysm which would significantly support a subsequent operation. To this end MRA data and DSA images are analyzed to extract information about the blood vessels such as position, orientation, width, and branchings. These items of information are input to the 3D reconstruction based on both imaging modalities. With these results physicians are able to in spect the relevant volume in three dimensions using various visualization tools including a combined display of MRA and the 3D DSA reconstruction. 2 The envisaged benefits of this system are reduced patient risk due to shorter DSA examinations involving less X-ray and contrast agent and improved representations of the pathology leading to a better diagnosis and treatment. This work is part of COVIRA (Computer Vision in Radiology), project A2003 of the AIM (Advanced Informatics in Medicine) programme of the European Commission.l This project started in 1992 and will last until 1994. 1. Introduction For many decades the radiologist's working tools were conventional X-ray and film. This has changed completely during the last decade as other imaging modalities such as Computer Tomography (CT), Magnetic Resonance Imaging (MRI), or Digital Subtraction Angiography (DSA) entered clinical practice. The number of images stemming from a single patient has enormously increased and makes the use of computers indispensable. This holds especially for visualiza tion where the display of three-dimensional data sets requires both sophisticated computer graphics and large computing resources. Far from being fully suffi cient the visualization tools supplied with an imaging system at least provide a basic functionality. But the radiologist wants more than just a separate display of the available imaging modalities. The latter provide him with different types of information about the same location in the human body, for example structural (anatomical) information from MRI and CT with functional information from Positron Emission Tomography (PET), or soft tissue anatomy from MRI with vessel anatomy from Magnetic Resonance Angiography (MRA) or DSA. This integra tion of information is still mainly left to the imagination of the radiologist who has to reduce a large amount of information to a concise diagnosis which he communicates to the physician re"ponsible for the patient's treatment. An inte- 1 Participants in the COVIRA consortium are: Philips; Medical Systems, Best (NL) (prime contractor) and Madrid (E); Corporate Research, Hamburg (D) Siemens AG, Erlangen (D) and Munich (D) IBM UK Scientific Centre, Winchester (UK) Gregorio Maranon General Hospital, Madrid (E) University of TUbingen, Neuroradiology and Theoretical Astrophysics (D) German Cancer Research Centre, Heidelberg (D) University of Leuven, Neurosurgery, Radiology and Electrical Engineering (B) University of Utrecht, Neurosurgery and Computer Vision (NL) Royal Marsden Hospital/Institute of Cancer Research, Sutton (UK) National Hospital for Neurology and Neurosurgery, London (UK) Foundation of Research and Technology, Crete (GR) University of Sheffield (UK) University of Genoa (I) University of Aachen (D) University of Hamburg (D) Federal Institute of Technology, ZUrich (CH)