Handbook of NEURAL NETWORK SIGNAL PROCESSING THE ELECTRICAL ENGINEERING AND APPLIED SIGNAL PROCESSING SERIES Edited by Alexander Poularikas The Advanced Signal Processing Handbook: Theory and Implementation for Radar, Sonar, and Medical Imaging Real-Time Systems Stergios Stergiopoulos The Transform and Data Compression Handbook K.R. Rao and P.C. Yip Handbook of Multisensor Data Fusion David Hall and James Llinas Handbook of Neural Network Signal Processing Yu Hen Hu and Jenq-Neng Hwang Handbook of Antennas in Wireless Communications Lal Chand Godara Forthcoming Titles Propagation Data Handbook for Wireless Communications Robert Crane The Digital Color Imaging Handbook Guarav Sharma Applications in Time Frequency Signal Processing Antonia Papandreou-Suppappola Noise Reduction in Speech Applications Gillian Davis Signal Processing in Noise Vyacheslav Tuzlukov Electromagnetic Radiation and the Human Body: Effects, Diagnosis, and Therapeutic Technologies Nikolaos Uzunoglu and Konstantina S. Nikita Digital Signal Processing with Examples in MATLAB® Samuel Stearns Smart Antennas Lal Chand Godara Pattern Recognition in Speech and Language Processing Wu Chou and Bing Huang Juang Handbook of NEUR AL NETWORK SIGNAL PROCESSING Edited by YU HEN HU JENQ-NENG HWANG CRC PR ESS Boca Raton London New York Washington, D.C. disclaimer Page 1 Wednesday, August 1, 2001 10:12 AM Library of Congress Cataloging-in-Publication Data Handbook of neural network signal processing / editors, Yu Hen Hu, Jenq-Neng Hwang. p. cm.— (Electrical engineering and applied signal processing (Series)) Includes bibliographical references and index. ISBN 0-8493-2359-2 1. Neural networks (Computer science)—Handbooks, manuals, etc. 2. Signal processing—Handbooks, manuals, etc. I. Hu, Yu Hen. II. Hwang, Jenq-Neng. III. Electrical engineering and signal processing series. QA76.87 H345 2001 006.3′2—dc21 2001035674 This book contains information obtained from authentic and highly regarded sources. Reprinted material is quoted with permission, and sources are indicated. A wide variety of references are listed. Reasonable efforts have been made to publish reliable data and information, but the author and the publisher cannot assume responsibility for the validity of all materials or for the consequences of their use. Neither this book nor any part may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, microfilming, and recording, or by any information storage or retrieval system, without prior permission in writing from the publisher. All rights reserved. Authorization to photocopy items for internal or personal use, or the personal or internal use of specific clients, may be granted by CRC Press LLC, provided that $1.50 per page photocopied is paid directly to Copyright clearance Center, 222 Rosewood Drive, Danvers, MA 01923 USA. The fee code for users of the Transactional Reporting Service is ISBN 0-8493-2359-2/01/$0.00+$1.50. The fee is subject to change without notice. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged. The consent of CRC Press LLC does not extend to copying for general distribution, for promotion, for creating new works, or for resale. Specific permission must be obtained in writing from CRC Press LLC for such copying. Direct all inquiries to CRC Press LLC, 2000 N.W. Corporate Blvd., Boca Raton, Florida 33431. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation, without intent to infringe. Visit the CRC Press Web site at www.crcpress.com © 2002 by CRC Press LLC No claim to original U.S. Government works International Standard Book Number 0-8493-2359-2 Library of Congress Card Number 2001035674 Printed in the United States of America 1 2 3 4 5 6 7 8 9 0 Printed on acid-free paper Preface Thefieldofartificialneuralnetworkshasmadetremendousprogressinthepast20yearsinterms of theory, algorithms, and applications. Notably, the majority of real world neural network appli- cationshaveinvolvedthesolutionofdifficultstatisticalsignalprocessingproblems. Comparedto conventionalsignalprocessingalgorithmsthataremainlybasedonlinearmodels, artificialneural networks offer an attractive alternative by providing nonlinear parametric models with universal approximation power, as well as adaptive training algorithms. The availability of such powerful modelingtoolsmotivatednumerousresearcheffortstoexplorenewsignalprocessingapplications ofartificialneuralnetworks. Duringthecourseoftheresearch,manyneuralnetworkparadigmswere proposed. Some of them are merely reincarnations of existing algorithms formulated in a neural network-likesetting, whiletheothersprovidenewperspectivestowardsolvingnonlinearadaptive signalprocessing. Moreimportantly,thereareanumberofemergentneuralnetworkparadigmsthat havefoundsuccessfulrealworldapplications. The purpose of this handbook is to survey recent progress in artificial neural network theory, algorithms (paradigms) with a special emphasis on signal processing applications. We invited a panelofinternationallywellknownresearcherswhohaveworkedonboththeoryandapplicationsof neuralnetworksforsignalprocessingtowriteeachchapter. Thereareatotalof12chaptersplusone introductorychapterinthishandbook. Thechaptersarecategorizedintothreegroups. Thefirstgroup containsin-depthsurveysofrecentprogressinneuralnetworkcomputingparadigms. Itcontainsfive chapters, including the introduction, that deal with multilayer perceptrons, radial basis functions, kernel-basedlearning,andcommitteemachines. Thesecondpartofthishandbooksurveystheneural networkimplementationsofimportantsignalprocessingproblems. Thispartcontainsfourchapters, dealing with a dynamic neural network for optimal signal processing, blind signal separation and blinddeconvolution,aneuralnetworkforprincipalcomponentanalysis,andapplicationsofneural networks to time series predictions. The third part of this handbook examines signal processing applications and systems that use neural network methods. This part contains chapters dealing withapplicationsofartificialneuralnetworks(ANNs)tospeechprocessing,learningandadaptive characterization of visual content in image retrieval systems, applications of neural networks to biomedicalimageprocessing,andahierarchicalfuzzyneuralnetworkforpatternclassification. The theory and design of artificial neural networks have advanced significantly during the past 20years. Muchofthatprogresshasadirectbearingonsignalprocessing. Inparticular,thenonlinear natureofneuralnetworks,theabilityofneuralnetworkstolearnfromtheirenvironmentsinsuper- visedand/orunsupervisedways,aswellastheuniversalapproximationpropertyofneuralnetworks makethemhighlysuitedforsolvingdifficultsignalprocessingproblems. Fromasignalprocessingperspective,itisimperativetodevelopaproperunderstandingofbasic neural network structures and how they impact signal processing algorithms and applications. A challengeinsurveyingthefieldofneuralnetworkparadigmsistodistinguishthoseneuralnetwork structuresthathavebeensuccessfullyappliedtosolverealworldproblemsfromthosethatarestill under development or have difficulty scaling up to solve realistic problems. When dealing with signalprocessingapplications,itiscriticaltounderstandthenatureoftheproblemformulationso thatthemostappropriateneuralnetworkparadigmcanbeapplied. Inaddition,itisalsoimportant to assess the impact of neural networks on the performance, robustness, and cost-effectiveness of signal processing systems and develop methodologies for integrating neural networks with other signalprocessingalgorithms. We would like to express our sincere thanks to all the authors who contributed to this hand- book: Michael T. Manry, Hema Chandrasekaran, and Cheng-Hsiung Hsieh (Chapter 2); Andrew D.Back(Chapter3);Klaus-RobertMüller,SebastianMika,GunnarRätsch,KojiTsuda,andBern- hardScholköpf(Chapter4);VolkerTresp(Chapter5);JoseC.Principe(Chapter6);ScottC.Douglas (Chapter 7); Konstantinos I. Diamantaras (Chapter 8); Yuansong Liao, John Moody, and Lizhong Wu(Chapter9);ShigeruKatagirig(Chapter10);PaisarnMuneesawang,Hau-SanWong,JoseLay, and Ling Guan (Chapter 11); Tülay Adali, Yue Wang, and Huai Li (Chapter 12); and Jinshiuh Taur,Sun-YuanKung,andShang-HungLin(Chapter13). Manyreviewershavecarefullyreadthe manuscript and provided many constructive suggestions. We are most grateful for their efforts. TheyareAndrewD.Back,DavidG.Brown,LaiwanChan,KonstantinosI.Diamantaras,Adriana Dumitras,MarkGirolami,LingGuan,KuldipPaliwal,AmandaSharkey,andJinshiuhTaur. Wewouldliketothanktheeditor-in-chiefofthisseriesofhandbooks,Dr.AlexanderD.Poularikas, forhisencouragement. OurmostsincereappreciationtoNoraKonopkaatCRCPressforherinfinite patienceandunderstandingthroughoutthisproject. Editors YuHenHureceivedaB.S.E.E.degreefromNationalTaiwanUniversity,Taipei,Taiwan,in1976. HereceivedM.S.E.E.andPh.D.degreesinelectricalengineeringfromtheUniversityofSouthern CaliforniainLosAngeles,in1980and1982,respectively. From1983to1987,hewasanassistant professorintheelectricalengineeringdepartmentofSouthernMethodistUniversityinDallas,Texas. HejoinedthedepartmentofelectricalandcomputerengineeringattheUniversityofWisconsinin Madison,asanassistantprofessorin1987,andheiscurrentlyanassociateprofessor. Hisresearch interestsincludemultimediasignalprocessing,artificialneuralnetworks,fastalgorithmsanddesign methodologyforapplicationspecificmicro-architectures,aswellascomputeraideddesigntoolsfor VLSIusingartificialintelligence. Hehaspublishedmorethan170technicalpapersintheseareas. His recent research interests have focused on image and video processing and human computer interface. Dr.HuisaformerassociateeditorforIEEETransactionsofAcoustic, Speech, andSignalPro- cessingintheareasofsystemidentificationandfastalgorithms. Heiscurrentlyassociateeditorof theJournalofVLSISignalProcessing. HeisafoundingmemberoftheNeuralNetworkSignalPro- cessingTechnicalCommitteeoftheIEEESignalProcessingSocietyandservedascommitteechair from1993to1996. HeisaformermemberoftheVLSISignalProcessingTechnicalCommitteeof theSignalProcessingSociety. Recently,heservedasthesecretaryoftheIEEESignalProcessing Society(1996–1998). Dr.HuisafellowoftheIEEE. Jenq-NengHwangholdsB.S.andM.S.degreesinelectricalengineeringfromtheNationalTaiwan University,Taipei,Taiwan. Aftercompletingtwoyearsofobligatorymilitaryservicesaftercollege, heenrolledasaresearchassistantattheSignalandImageProcessingInstituteofthedepartmentof electricalengineeringattheUniversityofSouthernCalifornia,wherehereceivedhisPh.D.degree inDecember1988. HewasalsoavisitingstudentatPrincetonUniversityfrom1987to1989. Inthesummerof1989,Dr.HwangjoinedtheDepartmentofElectricalEngineeringoftheUni- versity of Washington in Seattle, where he is currently a professor. He has published more than 150journalandconferencepapersandbookchaptersintheareasofimage/videosignalprocessing, computationalneuralnetworks,andmultimediasystemintegrationandnetworking. Hereceivedthe 1995IEEESignalProcessingSociety’sAnnualBestPaperAward(withShyh-RongLayandAlan Lippman)intheareaofneuralnetworksforsignalprocessing. Dr.HwangisafellowoftheIEEE.HeservedasthesecretaryoftheNeuralSystemsandApplica- tionsCommitteeoftheIEEECircuitsandSystemsSocietyfrom1989to1991,andhewasamember oftheDesignandImplementationofSignalProcessingSystemsTechnicalCommitteeoftheIEEE SignalProcessingSociety. HeisalsoafoundingmemberoftheMultimediaSignalProcessingTech- nicalCommitteeoftheIEEESignalProcessingSociety. HeservedasthechairmanoftheNeural NetworksSignalProcessingTechnicalCommitteeoftheIEEESignalProcessingSocietyfrom1996 to1998,andheiscurrentlytheSociety’srepresentativetotheIEEENeuralNetworkCouncil. He served as an associate editor for IEEE Transactions on Signal Processing from 1992 to 1994 and currentlyistheassociateeditorforIEEETransactionsonNeuralNetworksandIEEETransactions on Circuits and Systems for Video Technology. He is also on the editorial board of the Journal of VLSISignalProcessingSystemsforSignal,Image,andVideoTechnology. Dr. Hwangwasthecon- ferenceprogramchairofthe1994IEEEWorkshoponNeuralNetworksforSignalProcessingheldin Ermioni,GreeceinSeptember1994. Hewasthegeneralco-chairoftheInternationalSymposiumon ArtificialNeuralNetworksheldinHsinchu,TaiwaninDecember1995. Healsochairedthetutorial committeefortheIEEEInternationalConferenceonNeuralNetworksheldinWashington,D.C.in June1996. Hewastheprogramco-chairoftheInternationalConferenceonAcoustics,Speech,and SignalProcessinginSeattle,Washingtonin1998. 2359/Contributors Page i Thursday, August 2, 2001 12:52 PM Contributors Tülay Adali Sun-Yuan Kung Jose C. Principe University of Maryland Princeton University University of Florida Baltimore, Maryland Princeton, New Jersey Gainesville, Florida Andrew D. Back Jose Lay Gunnar Rätsch Windale Technologies University of Sydney GMD FIRST and University of Brisbane, Australia Sydney, Australia Potsdam Berlin, Germany Hema Chandrasekaran Huai Li U.S. Wireless Corporation University of Maryland Bernhard Schölkopf San Ramon, California Baltimore, Maryland Max-Planck-Institut für Biologische Kybernetik Konstantinos I. Yuansong Liao Tübingen, Germany Diamantaras Oregon Graduate Institute of Technological Education Institute Science and Technology Junshiuh Taur of Thessaloniki Beaverton, Oregon National Chung-Hsing University Sindos, Greece Taichung Shang-Hung Lin Taiwan, China Scott C. Douglas EPSON Palo Alto Laboratories Southern Methodist University ERD Volker Tresp Dallas, Texas Palo Alto, California Siemens AG Corporate Technology Ling Guan Michael T. Manry Munich, Germany University of Sydney University of Texas Sydney, Australia Arlington, Texas Koji Tsuda AIST Computational Biology Cheng-Hsiung Hsieh Sebastian Mika Research Center Chien Kou Institute of Technology GMD FIRST Tokyo, Japan Changwa Berlin, Germany Taiwan, China Yue Wang John Moody The Catholic Universtiy of Yu Hen Hu Oregon Graduate Institute of America University of Wisconsin Science and Technology Washington, DC Madison, Wisconsin Beaverton, Oregon Hau-San Wong Jenq-Neug Hwang Klaus-Robert Müler University of Sydney University of Washington GMD FIRST and University of Sydney, Australia Seattle, Washington Potsdam Berlin, Germany Lizhong Wu Shigeru Katagiri HNC Software, Inc. Intelligent Communication Science Paisarn Muneesawang San Diego, California Laboratories University of Sydney Kyoto, Japan Sydney, Australia