Maximum-Entropy and Bayesian Spectral Analysis and Estimation Problems Fundamental Theories of Physics A New International Book Series on the Fundamental Theories of Physics: Their Clarification, Development and Application Editor: ALWYN VAN DER MERWE University of Denver, U.S.A. Editorial Advisory Board: ASIM BARUT, University of Colorado, U.S.A. HERMANN BONDI, Natural Environment Research Council, U.K. BRIAN D. JOSEPHSON, University of Cambridge, U.K. CLIVE KILMISTER, University of London, U. K. GONTER LUDWIG, Philipps-Universitiit, Marburg, F.R.G. NATHAN ROSEN, Israel Institute of Technology, Israel MENDEL SACHS, State University of New York at Buffalo, U.S.A. ABDUS SALAM, International Centre for Theoretical Physics, Trieste, Italy HANS-JORGEN TREDER, Zentralinstitut fur Astrophysik der Akademie der Wissenschaften, G.D.R. Maximum-Entropy and Bayesian Spectral Analysis and Estimation Problems Proceedings of the Third Workshop on Maximum Entropy and Bayesian Methods in Applied Statistics, Wyoming, U.S.A., August 1-4, 1983 edited by c. Ray Smith u.s. Army Missile Command, Redstone Arsenal, Alabama, U.S.A. and Gary J. Erickson Department a/Electrical Engineering, Seattle University, Seattle, Washington. U.S.A. D. Reidel Publishing Company A MEMBER OF THE KLUWER ACADEMIC PUBLISHERS GROUP Dordrecht / Boston / Lancaster / Tokyo Libnry of Congress Cataloging in Public:ation Data Maximum Entropy Workshop (3rd: 1983: Laramie, Wyo.) Maximum-entropy and Bayesian spectral analysis and estimation problems. (Fundamental theories of physics) Includes index. 1. Entropy (Information theory)-Congresses. 2. Bayesian statistical decision theory-Congresses. I. Smith, C. Ray, 1933- . II. Erickson, Gary J. III. Title. IV, Series. Q370.M385 1983 001.53'9 87-23228 ISBN-13: 978-94-010-8257-0 e-ISBN-13: 978-94-009-3961-5 DOl: 10.1007/978-94-009-3961-5 Published by D. Reidel Publishing Company, P.O. Box 17, 3300 AA Dordrecht, Holland. Sold and distributed in the U.S.A. and Canada by Kluwer Academic Publishers, 101 Philip Drive, Assinippi Park, Norwell, MA 02061, U.S.A. In all other countries, sold and distributed by Kluwer Academic Publishers Group, P.O. Box 322,3300 AH Dordrecht, Holland. All Rights Reserved © 1987 by D. Reidel Publishing Company, Dordrecht, Holland Softcover reprint of the hardcover 1st edition 1987 No part of the material protected by this copyright notice may be reproduced or utilized in any form or by any means, electronic or mechanical including photocopying, recording or by any information storage and retrieval system, without written permission from the copyright owner To the memories of our fathers, Robert Austin Smith and Phillip Christian Erickson CONTENTS Preface •• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• ix BAYESIAN SPECTRUM AND CHIRP ANALYSIS E. T. Jaynes ••••••••••••••••••••••••••••••••••••• 1 ON ENTROPY RATE Athanasios Papoulis •••••••••••••••••••••••••••••••• 39 STATE SPACES AND INITIAL ESTIMATES IN MINIMUM RELATIVE-ENTROPY INVERSION WITH APPLICATION TO SPECTRUM ANALYSIS AND IMAGE ENHANCEMENT John E. Shore •••••••••••••••••••••••••••••••••••• 51 RELATIVE-ENTROPY MINIMIZATION WITH UNCERTAIN CONSTRAINTS: THEORY AND APPLICATION TO SPECTRUM ANALYSIS Rodney W. Johnson • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• 57 A PROOF OF BURG'S THEOREM B. S. Choi and Thomas M. Cover •••••••••••••••••••••••• 75 A BAYESIAN APPROACH TO ROBUST LOCAL FACET ESTIMATION Robert M. Haralick • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• 85 THE MAXIMUM ENTROPY METHOD: THE PROBLEM OF MISSING DATA William I. Newman • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• 99 ON THE ACCURACY OF SPECTRUM ANALYSIS OF RED NOISE PROCESSES USING MAXIMUM ENTROPY AND PERIODOGRAM METHODS: SIMULATION STUDIES AND APPLICATION TO GEOPHYSICAL DATA Paul F. Fougere. • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• 127 RECENT DEVELOPMENTS AT CAMBRIDGE Stephen F. Gull and John Skilling 149 PRIOR KNOWLEDGE MUST BE USED ...................... John Skilling and Stephen F. Gull 161 HOW THE BRAIN WORKS: THE NEXT GREAT SCIENTIFIC REVOLUTION David Hestenes •••••••••••••••••••••••••••••••••• 173 viii CONTENTS MAXIMUM ENTROPY IN STRUCTURAL MOLECULAR BIOLOGY: THE FIBER DIFFRACTION PHASE PROBLEM Richard K. Bryan ••••••••••••••••••••••••••••••••• 207 A METHOD OF COMPUTING MAXIMUM ENTROPY PROBABILITY VALUES FOR EXPERT SYSTEMS Peter Cheeseman ••••••••••••••••••••••••••••••••• 229 SPECIAL-PURPOSE ALGORITHMS FOR LINEARLY CONSTRAINED ENTROPY MAXIM IZATION Yair Censor, Tommy Elfving, and Gabor T. Herman ••••••••••• 241 BAYESIAN APPROACH TO LIMITED-ANGLE RECONSTRUCTION IN COMPUTED TOMOGRAPHY Kenneth M. Hanson and George W. Wecksung •• • • • • • • • • • • • •• 255 APPLICATION OF THE MAXIMUM ENTROPY PRINCIPLE TO RETRIEVAL FROM LARGE DATA BASES Paul B. Kantor •••••••••••••••••••••••••••••••••• 273 TWO RECENT APPLICATIONS OF MAXIMUM ENTROPY Lee H. Schick ••••••••••••••••••••••••••••••••••• 283 A VARIATIONAL METHOD FOR CLASSICAL FLUIDS Ramarao Inguva, C. Ray Smith, T. M. Huber, and Gary Erickson. •• 295 UPDATING INDUCTIVE INFERENCE N. C. Dalkey ••••••••••••••••••••••••••••••••••• 305 PARALLEL ALGORITHMS FOR MAXIMUM ENTROPY CALCULATION Stuart Cernan • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• 317 Subject Index ••••••••••••••••••••••••••••••••••••• 319 PREFACE This volume has its origin in the third ·Workshop on Maximum-Entropy and Bayesian Methods in Applied Statistics,· held at the University of Wyoming, August 1 to 4, 1983. It was anticipated that the proceedings of this workshop could not be prepared in a timely fashion, so most of the papers were not collected until a year or so ago. Because most of the papers are in the nature of advancing theory or solving specific problems, as opposed to status reports, it is believed that the contents of this volume will be of lasting interest to the Bayesian community. The workshop was organized to bring together researchers from differ ent fields to examine critically maximum-entropy and Bayesian methods in science, engineering, medicine, economics, and other disciplines. Some of the papers were chosen specifically to kindle interest in new areas that may offer new tools or insight to the reader or to stimulate work on pressing problems that appear to be ideally suited to the maximum-entropy or Bayes ian method. Certain facets of publishing a book are inherently unrewarding and frus trating. Or so it seems until the task is completed, and one has the pleasure of acknowledging publicly those who have helped along the way. Adequate thanks to Martha Stockton are impossible. The camera-ready copy prepared by Martha has benefited substantially by her editorial, proofreading, and drafting assistance. Dr. David Larner and Professor Alwyn van der Merwe, both affiliated with Reidel, provided encouragement and friendship at criti cal times. We are happy that Reidel has agreed to publish future proceed ings of these workshops. Others who have made our work easier or more rewarding include Evelyn Haskell, Marce Mitchum, and our friends of the SDC Passive Sensors Division. Dr. Rabinder Madan of the Office of Naval Research has provided continual encouragement and assisted us in obtaining much-needed funding. August 1987 C. Ray Smith Gary J. Erickson ix BAYESIAN SPECTRUM AND CHIRP ANALYSIS E. T. Jaynes Wayman Crow Professor of Physics Washington University, St. Louis, MO 63130 We seek optimal methods of estimating power spectrum and chirp (fre quency change) rate for the case that one has incomplete noisy data on values y(t) of a time series. The Schuster periodogram turns out to be a ·sufficient statistic· for the spectrum, a generalization playing the same role for chirped signals. However, the optimal processing is not a linear filtering operation like the Blackman-Tukey smoothing of the periodogram, but rather a nonlinear operation. While suppressing noise/side lobe arti facts, it achieves the same kind of improved resolution that the Burg method did for noiseless data. C. R. Smith and G. J. Erickson (eds.), Maximum-Entropy and Bayesian Spectral Analysis and Estimation Problems, 1-37. © 1987 by D. Reidel Publishing Company.