DATA FUSION FOR SENSORY INFORMATION PROCESSING SYSTEMS THE KLUWER INTERNATIONAL SERIES IN ENGINEERING AND COMPUTER SCIENCE ROBOTICS: VISION, MANIPULATION AND SENSORS Consulting Editor Takeo Kanade Other books in the series: ROBOTIC GRASPING AND FINE MANIPULATION, M. Cutkosky ISBN: 0-89838-200-9 SHADOWS AND SILHOUETfES IN COMPUTER VISION, S. Shafer ISBN: 0-89838-167-3 PERCEPlUAL ORGANIZATION AND VISUAL RECOGNITION, D. Lowe ISBN: 0-89838-172-X ROBOT DYNAMICS ALGORTIHMS, F. Featherstone ISBN: 0-89838-230-0 TIlREE-DIMENSIONAL MACHINE VISION, T. Kanade (editor) ISBN: 0-89838-188-6 KINEMATIC MODEliNG, IDENTIFICATION AND CONTROL OF ROBOT MANIPULATORS, H.W. Stone ISBN: 0-89838-237-8 OBJECT RECOGNITION USING VISION AND TOUCH, P. Allen ISBN: 0-89838-245-9 INTEGRATION, COORDINATION AND CONTROL OF MULTI-SENSOR ROBOT SYSTEMS, H.F. Durrant-Whyte ISBN: 0-89838-247-5 MOTION UNDERSTANDING: Robot and Human Vision, W.N. Martin and J. K. Aggrawal (editors) IsBN: 0-89838-258-0 BAYESIAN MODELING OF UNCERTAINTY IN LOW-LEVEL VISION, R.Szeliski ISBN 0-7923-9039-3 VISION AND NAVIGATION: TIlE CMU NAV LAB, C. Thorpe (editor) ISBN 0-7923-9068-7 TASK-DIRECI'ED SENSOR FUSION AND PLANNING: A Computational Approach, G. D. Hager ISBN: 0-7923-9108-X COMPUTER ANALYSIS OF VISUAL TEXTURES, F. Tomita and S. Tsuji ISBN: 0-7923-9114-4 DATA FUSION FOR SENSORY INFORMATION PROCESSING SYSTEMS by James J. Clark Alan L. Yuille Division of Applied Sciences Harvard University Cambridge, Massachusetts ~. " SPRINGER SCIENCE+BUSINESS MEDIA, LLC Library of Congress Cataloging-in-Publication Data Clark, James Joseph, 1957- Data fusion for sensory information processing systems / James J. Clark, Alan L. Yuille. p. cm. - Kluwer international series in engineering and computer science ; SECS 105) Includes bibliographical references (p. ) and index. ISBN 978-1-4419-5126-7 ISBN 978-1-4757-2076-1 (eBook) DOI 10.1007/978-1-4757-2076-1 1. Computer vision. 2. Image processing. I. Yuille, A. L. (Alan L.) II. Title. III. Series. TA1632.C58 1990 006.3'7-dc20 90-4770 CIP Copyright © 1990 by Springer Science+Business Media New York Originally published by Kluwer Academic Publishers in 1990 Softcover reprint of the hardcover 1st edition 1990 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, mechanical, photo-copying, recording, or otherwise, without the prior written permission of the publisher, Springer Science+Business Media, LLC. Contents Preface xiii 1 Introduction: The Role of Data Fusion In Sensory Systems 1 1.1 INFORMATION ACQUISITION: INVERTING THE WORLD-IMAGE MAPPING . . . . . . . . . . . . .. 1 1.2 THE NEED FOR CONSTRAINTS. . . . 5 1.3 DETERMINATION AND EMBEDDING OF CONSTRAINTS . . . . . . . . . . . . . . . . . . . . 9 1.4 THE NEED FOR DATA FUSION .... 13 1.5 SUMMARy........................ 15 2 Bayesian Sensory Information Processing 17 2.1 BAYES RULE ............. 18 2.2 THE IMAGE FORMATION MODEL 19 v vi 2.3 THE PRIORS ...................... 23 2.3.1 THE SYSTEM MODEL. . . . . . . . . . . .. 24 f .. . . . . . . .. 2.4 BAYESIAN ESTIMATORS FOR 25 2.5 BAYESIAN DETECTION AND EXTRACTION SYSTEMS ......................... 28 2.6 THE BAYESIAN CONTROVERSY . . . . . . . . .. 30 2.7 CHAPTER S'UMMARY . . . . . . . . . . . . . . . .. 37 3 Information Processing U sing Energy Function Minimization ' 39 3.1 MARKOV RANDOM FIELDS . . . . . . . . . . . .. 44 3.2 ENERGY FUNCTIONS WITH MATCHING ELEMENTS ....................... 46 3.3 STATISTICAL MECHANICS AND MEAN FIELD THEORY ......................... 50 3.3.1 W.T.A. WITH THE MEAN FIELD APPROXIMATION . . . . . . . . . . . . . .. 51 3.3.2 W.T.A. WITHOUT THE MEAN FIELD APPROXIMATION . . . . . . . . . . . . . .. 53 3.3.3 AVERAGING OUT FIELDS . . . . . . . . .. 54 3.4 THE FORM OF THE SMOOTHNESS CONSTRAINT 55 3.5 ALTERNATIVE FORMS OF CONSTRAINT. . . .. 57 VB 3.5.1 PARAMETRIC CONSTRAINTS ...... " 57 3.5.2 MINIMAL DESCRIPTION LENGTH CODING 64 3.5.3 MULTIPLE SETS OF PRIORS . 66 3.6 CHAPTER SUMMARY ......... . 68 4 Weakly vs. Strongly Coupled Data Fusion: A Classification of Fusional Methods 71 4.1 A CLASSIFICATION 0F FUSIONAL METHODS. 72 4.2 WEAKLY COUPLED DATA FUSION ......... 73 4.2.1 CLASS I WEAKLY COUPLED DATA FUSION 73 4.2.2 CLASS II WEAKLY COUPLED DATA FUSION ...................... 75 4.2.3 CLASS III WEAKLY COUPLED DATA FUSION ............... ....... 76 4.3 STRONGLY COUPLED DATA FUSION ALGORITHMS ............... ....... 78 4.3.1 STRONG COUPLING BY PRIOR CONSTRAINT ADAPTION ................... 80 4.3.2 STRONG COUPLING BY ADAPTION OF THE IMAGE FORMATION MODEL ..... 80 4.3.3 RECURRENT STRONG COUPLING ..... 81 4.3.4 COUPLED MRF METHODS AS STRONGLY COUPLED DATA FUSION ........... 82 Vlll 4.4 BAYESIAN IMPLEMENTATION OF DATA FUSION 83 4.5 EXAMPLES OF WEAKLY COUPLED DATA FUSION IN THE VISION LITERATURE. . . . . .. 87 4.6 EXAMPLES OF STRONGLY COUPLED FUSION IN THE VISION LITERATURE . . . . . . . .. 91 4.7 SUMMARy............ · 103 5 Data Fusion Applied to Feature Based Stereo Algorithms 105 5.1 INTRODUCTION .......... 105 5.2 THE BAYESIAN APPROACH TO STEREO VISION 108 5.2.1 THE MATCHING PROBLEM ......... 108 5.2.2 THE FIRST LEVEL: MATCHING FIELD AND DISPARITY FIELD ............... 109 5.2.3 THE SECOND LEVEL: ADDING DISCONTINUITY FIELDS . · 112 5.2.4 THE THIRD LEVEL: ADDING INTENSITY TERMS ...................... 113 5.2.5 THE BAYESIAN FORMULATION OF THE STEREO ALGORITHM ............ 114 5.3 STATISTICAL MECHANICS AND MEAN FIELD THEORY. . . . . . . . . . . . . . . . ...... 116 5.3.1 AVERAGING OUT FIELDS · 117 ix 5.3.2 DETERMINISTIC SOLUTIONS OF THE MEAN FIELD EQUATIONS .............. 125 5.4 COMPARISONS WITH OTHER THEORIES ..... 128 5.4.1 THE MARR-POGGIO COOPERATIVE STEREO ALGORITHM ............ 128 5.4.2 DISPARITY GRADIENT LIMIT THEORIES 130 5.5 COMPARISONS WITH PSYCHOPHYSICAL DATA 131 5.6 CHAPTER SUMMARY . . . . . . . . . . . . . . . . . 134 6 Fusing Binocular and Monocular Depth Cues 137 6.1 STRONG FUSION STEREO WITH MONOCULAR CUES .................. 137 6.2 PREVIOUS ATTEMPTS AT STRONG COUPLING FOR STEREO ...................... 138 6.2.1 A GENERAL FRAMEWORK ......... 142 6.2.2 SOFT AND HARD CONSTRAINTS ..... 144 6.3 SUMMARy ........................ 146 7 Data Fusion in Shape From Shading Algorithms 147 7.1 AN ALGEBRAIC APPROACH TO FUSING SPECULAR AND LAMBERTIAN REFLECTANCE DATA ........................... 148 x 7.2 A CLASS III WEAKLY COUPLED FUSION IMPLEMENTATION .................. 158 7.3 A STRONGLY COUPLED APPROACH TO POLYCHROMATIC SHAPE FROM SHADING ... 168 7.4 FUSION OF IMAGE FORMATION MODELS .... 172 7.5 CHAPTER SUMMARY . . . . . . . . . . . . . . . . . 179 8 Temporal Aspects of Data Fusion 181 8.1 A TEMPORAL COHERENCE EDGE DETECTOR. 182 8.1.1 DETERMINATION OF THE CONDITIONAL DENSITIES .................... 185 8.1.2 BAYESIAN EDGE DETECTION DECISION PROCESS .................... 189 8.2 A STRONGLY COUPLED TEMPORAL COHER- ENCE EDGE DETECTOR ............... 197 8.3 TEMPORAL SAMPLING ................ 201 8.3.1 COMPUTATIONAL CONSTRAINTS ..... 204 8.4 ACTIVE DETERMINATION OF CONSTRAINTS . 206 8.5 SUMMARy........................ 215