Akira Hirose Complex-Valued Neural Networks Second Edition 123 Author Prof. Akira Hirose The University of Tokyo Japan Original Japanese Language Edition Published by Saiensu-shaCo., Ltd. 1-3-25, Sendagaya, Shibuya-ku,Tokyo 151-0051, Japan Copyright 2005, Saiensu-sha Co., Ltd.All Rights Reserved ISSN1860-949X e-ISSN1860-9503 ISBN 978-3-642-27631-6 e-ISBN 978-3-642-27632-3 DOI 10.1007/978-3-642-27632-3 Springer Heidelberg New YorkDordrecht London LibraryofCongressControlNumber:2011945320 (cid:2)c Springer-VerlagBerlinHeidelberg2012 Preface to the Second Edition Five years have passed since the First Edition of this Book was published. Fortunately it obtained a large readership, and seems to have made a hum- ble contribution to the development of the complex-valued neural-network research.The author is extremely grateful to the readersfor their explicit or implicit insightful commentsand continuoussupport. It is my greatpleasure to have the occasion to publish the Second Edition. During this period, the research on complex-valued neural networks ex- pands largely in both the quality and the quantity. There have been many special sessions in representative conferences which cover not only mathe- matical sciences but also electrical and electronic engineering such as the International Joint Conference on Neural Networks (IJCNN) sponsored by IEEEComputationalIntelligenceSociety(IEEECIS)andInternationalNeu- ral Network Society (INNS), including the biyearly held World Congress on ComputationalIntelligence(WCCI), the InternationalConferenceonNeural InformationProcessing(ICONIP)promotedby Asia-PacificNeural Network Assembly (APNNA) organized by Asia-Pacific countries’ neural network so- cieties such as Japanese Neural Network Society (JNNS), and the Interna- tional Conferece on Artificial Neural Networks (ICANN) organized by the EuropeanNeural Network Society (ENNS). A projectnamed”PracticalAp- plications of Complex-Valued Neural Networks” has also been organized as one of the Nation-wide Cooperative Research Projects in Research Institute of Electrical Communication (RIEC), Tohoku University, in 2009 to accel- erate application-oriented research in technology and society. A worldwide network, namely, the Task Force on Complex-Valued Neural Networks, has alsobeeninauguratedintheInstituteofElectricalandElectronicsEngineers (IEEE) Computational Intelligence Society (CIS) Neural Network Technical Committee (NNTC) with more than 40 active members. Recent progress is summarized as follows. The application fields spread more and more in electromagnetic-wave, lightwave and ultrasonic-wave engineering areas such as earth and environmental observation with satel- lite/airborne radar systems, security imaging at airports, railway stations and other public transportation, and medical diagnosis and monitoring uti- lizingcoherentwavephenomena.Othergrowingfieldsincludeadaptiveimage processinginfrequencydomainandtime-sequentialsignalprocessingwidely. Theoriesaregoingtoextendfromcomplexandquaternionnetworksto more general frameworks based on Clifford algebra. This Second Edition is anenlargedand revisedbook featuring mainly the followingtwopoints.Thefirstistheinclusionofrecenttrendsintheoverview in Section 3.7 (research history) as well as in respective chapter citation in PartII (applications), resultinginalmostdoublednumberofreferences.The parametroninventedin1954isalsoreferredtowithdiscussiononanalogyand disparity. The second point is the addition of argument on the advantages of the complex-valued neural networks in Sections 3.1 (history of complex number), 3.2 (degree of freedom and circularity in learning), 3.4 (metric in complex domain) and some other sections to enhance the difference from real-valued neural networks. TheauthorisverymuchobligedtoProf.JanuszKacprzyk,Editor-in-Chief, Studies inComputational Intelligence, andDr. ThomasDitzinger, Engineer- ingEditor,Springer-Verlag,fortheircontinuoushelpinpublication.Iexpress my sincere thanks alsoto Mr. KosukeHirase andMr. Nobuhiko Tajima, Ex- ecutive Editors, Saiensu-sha, for their kind permission and promotion. Tokyo, Japan Akira Hirose October 2011 Preface to the English First Edition The originalJapaneseedition ofthis book, published by Saiensu-sha,Japan, inMarch2004,hasfortunatelyacquiredafavorablereputation.Iamgrateful to the readers for their kind feedbacks, many of which are included in this edition. I hope this English publication attracts readers in wider areas, and evokes valuable feedbacks furthermore. In the months after the Japanese publication, researches on the complex- valued neural networks have kept evolution in respective directions. There are some plans of special sessions in international conferences and special issuesinjournals.Thebibliographyhas beenslightlymodifiedtoinclude the specialsessionsandlatestjournalpublications.Ontheotherhand,references written in Japanese on domestic circumstances have been omitted. Besides, Fig.1.1has beenadded, andFig.2.1has beenmodified, whichare relatedtotheSpecialIssueonComplex-ValuedNeuralNetworks,TheJournal oftheIEICE,87(6),June2004(inJapanese),sothatevenreadersnothaving glancedtheissuecanobtainclearconcepts.Withthesemodifications,Iexpect a higher appeal in this English edition. I am very much obliged to Dr. Thomas Ditzinger, Engineering Editor, Springer-Verlag, for his continuous help in publication. Tokyo, Japan Akira Hirose April 2006 Preface to the Japanese First Edition The studies on complex-valued neural networks have recently been evolv- ing in various directions. The pioneering areas include electromagnetic-wave andlightwavesensingandimaging,independentcomponentanalysisinblind separation, blur restoration in image processing, and so on. Developing ap- plications involve adaptive quantum devices, quantum computation, social systems related to periodicity and oscillation, and so forth. This book is the first one to describe systematically the complex-valued neuralnetworks.Itisrecommendableforresearchers,graduatestudents,and undergraduate students in, for example, electrical and electronic engineer- ing, informatics, control engineering, mechanics, robotics, and bioengineer- ing. This book is useful for those who begin to study, for instance, adaptive signal processing for highly functional sensing and imaging, control in un- knownandchangingenvironment,brain-likeinformationprocessing,robotics inspired by human neural systems, and interdisciplinary studies to realize comfortable society. It is also helpful to those who carry out research and development regarding new products and services at companies. The author wrotethisbookhopinginparticularthatthebookprovidesthemmeaningful hints to make good use of neural networks in fully practical applications. Thefirstmultiauthorbookfocusingonthecomplex-valuedneuralnetworks is “Complex-ValuedNeural Networks:Theories andApplications” published byWorldScientificPublishingCo.(October2003),inwhichresearchersmost active in this field reconstruct and present their pioneering works. Besides, a foreword describes the relation of the complex-valued networks with other fields and the real world. In addition, an introductory chapter outlines this widely expanding field listing theoretical and application aspects. On the other hand, this book, “Complex-Valued Neural Networks,” de- scribes the field systematically by a single researcher. The book emphasizes basicideasandwaysofthinking.Whydoweneedtoconsiderneuralnetworks that deal with complex numbers? What advantages do the complex-valued neural networks have? What is the origin of the advantages? In what areas do they develop principal applications? This book answers these questions by describing details and examples, which will inspire many readers with new ideas. It may not be completely comprehensive. Instead, the author de- cided to make effort not to include all things relevant to this field, but to describe vividly this extensivelydevelopingfield. Therefore,the fields, where thestrongpointsofthe complex-valuedneuralnetworkswill potentiallyplay important roles, extend further beyond those explained in this book. The book consists of two parts. PartI describes basic concepts, ideas and fundamentals,whilePartIIpresentsapplicationexamplesandillustrateshow to use the networks in specific situations. In Part I, an application example is presented in Chapter 1, so that the readers grasp the rough idea of the complex-valued neural networks. Chapter 2 provides briefly the viewpoint of the artificial neural networks in general in the framework of information processing. Chapter 3 presents basic ideas, overview of applications, and a historicoutline.Then, Chapter4explainsconstructionsanddynamicsofthe complex-valuedneuralnetworks.Inthedescriptions,conventionalneuralnet- works are firstpresented, and then,they are extendedintothe complex-valued networks. Consequently, the readers are able to understand the contents even if they do not have any knowledge on conventional neural networks. Therefore, the book is recommendable as an introduction not only to the complex-valued networks, but also widely to the artificial neural networks. Themannerofdescriptioninthisbookplacesstressontheintuitionalunder- standingofactualdynamics,practicaladvantages,andeffectiveapplications, rather than mathematical rigorousness. If the readers require formal mathe- matics further more, please refer to literature presented in each chapter. The fields relevantto the complex-valuedneural networksare enormously wide since, from a certain viewpoint, we may regard them as a superset of the conventional networks. The author is afraid that the book may includes serious errors and misunderstanding, on which the readers’ corrections√and suggestions are heartily welcome. Please note that the imaginary unit −1 is expressed as i or j in accordance with the customs in respective fields. The application examples presented in Part II have been obtained in the A. HiroseLaboratory,TheUniversityofTokyo,withmanygraduateandun- dergraduate students belonging to the Courses in Electrical and Electronic Engineering, Computer Science, Frontier Informatics, and Interdisciplinary Science and Technology. That is, the results in Chapters 5 (land-surface segmentation) and 7 (digital elevation map) have been obtained mainly by Dr. Andriyan Bayu Suksmono, Chapter 6 (landmine visualization radar system) by Mr. Takahiro Hara, Chapters 8 (carrier-frequency-dependent as- sociative memory) and 9 (optical phase equalizer) by Dr. Sotaro Kawata, Chapter10(developmentallearning)byMr.YasufumiAsanoandMr.Toshi- hiko Hamano, and Chapter 11 (voice synthesis) by Mr. Keiichi Tsuda. The authorexpresseshissincerethankstothemandallotherlaboratorymembers and alumni. The details in the studies are available on the web pages of the Laboratory (http://www.eis.t.u-tokyo.ac.jp/). The author is grateful also to Mr. Kosuke Hirase at Saiensu-sha, Japan, and Mr. Naoshi Takeda at Suuri-kogaku-sha, Japan, for their great help in publication. Lastly,the authorwouldliketoaskallthereadersfortheircooperationin this developing field by quoting from the Preface, contributed by Dr. Nobuo Hataoka, in the Special Issue on “Complex-Valued Neural Networks,” The JournaloftheIEICE,vol.87,No.6(2004)p.446:“Thecomplex-valuedneural networks deal with phase information in addition. They possess remarkable ability in comparison with the conventional networks. However, they have just departed from the starting point.” The author would like to make fur- ther effort to construct practically useful neural networks by obtaining the cooperation of researchersat large in the world. Tokyo, Japan Akira Hirose December 2004 Contents Part I Basic Ideas and Fundamentals: Why Are Complex-Valued Neural Networks Inevitable? 1 Complex-Valued Neural Networks Fertilize Electronics ... 3 1.1 Imitate the Brain, and Surpass the Brain.................. 3 1.2 Create a “Superbrain” by Enrichment of the Information Representation......................................... 4 1.3 Application Fields Expand Rapidly and Steadily ........... 6 1.4 Book Organization ..................................... 8 2 Neural Networks: The Characteristic Viewpoints ......... 9 2.1 Brain, Artificial Brain, Artificial Intelligence (AI), and Neural Networks ....................................... 9 2.2 Physicality of Brain Functions ........................... 12 2.3 Neural Networks: General Features ....................... 13 3 Complex-Valued Neural Networks: Distinctive Features .. 17 3.1 What Is a Complex Number? ............................ 17 3.1.1 Geometric and Intuitive Definition.................. 17 3.1.2 Definition as Ordered Pair of Real Numbers ......... 18 3.1.3 Real 2×2 Matrix Representation ................... 19 3.2 Comparison of Complex- and Real-Valued Feedforward Neural Networks ....................................... 20 3.2.1 Function of Complex-Valued Synapse and Network Operation ....................................... 20 3.2.2 Circularity and Widely-Linear Systems.............. 23 3.2.3 Nonlinearity That Enhances the Features of Complex-Valued Networks......................... 24 3.3 Activation Functions in Neurons.......................... 26 3.3.1 Nonlinear Activation Functions in Real-Valued Neural Networks ................................. 26 3.3.2 Problem Concerning Activation Functions in Complex-Valued Neural Networks .................. 28 3.3.3 Construction of CVNNs with Partial Derivatives in Complex Domain................................. 28 3.3.4 Real-Imaginary-Type Activation Function ........... 31 3.3.5 Amplitude-Phase-Type Activation Function ......... 32 3.4 Metric in Complex Domain .............................. 34 3.4.1 Importance of Metric: An Example in Complex-Valued Self-organizing Map ............... 34 3.4.2 Euclidean Metric ................................. 34 3.4.3 Complex-Valued Inner-Product Metric .............. 36 3.4.4 Comparison between Complex-Valued Inner Product and Euclidean Distance ........................... 36 3.4.5 Metric in Correlation Learning..................... 37 3.5 What Is the Sense of Complex-Valued Information and Its Processing?............................................ 38 3.6 In What Fields Are CVNNs Effective? .................... 40 3.6.1 Electromagnetic and Optical Waves, Electrical Signals in Analog and Digital Circuits............... 40 3.6.2 Electron Wave ................................... 43 3.6.3 Superconductors ................................. 45 3.6.4 Quantum Computation ........................... 45 3.6.5 Sonic and Ultrasonic Waves........................ 45 3.6.6 Compatibility of Controllability and Adaptability..... 46 3.6.7 Periodic Topology and Metric...................... 46 3.6.8 Direct Use of Polar Coordinates.................... 48 3.6.9 High Stability in Recurrent Dynamics............... 48 3.6.10 Preservationof Relative Directions and Segmentation Boundaries in Two-Dimensional Information Transform....................................... 49 3.6.11 Chaos and Fractals in Complex Domain............. 49 3.6.12 Quaternion and Other Higher-Order Complex Numbers ........................................ 49 3.7 Investigations in Complex-Valued Neural Networks ......... 50 3.7.1 History ......................................... 50 3.7.2 Recent Progress.................................. 53 4 Constructions and Dynamics of Neural Networks ......... 57 4.1 Processing, Learning, and Self-organization ................ 57 4.1.1 Pulse-Density Signal Representation ................ 57 4.1.2 Neural Dynamics................................. 59 4.1.3 Task Processing.................................. 59 4.1.4 Learning and Self-organization ..................... 60 4.1.5 Changes in Connection Weights .................... 60 4.2 Hebbian Rule .......................................... 60