Part I Basic Ideas and Fundamentals: Why are complex-valued neural networks inevitable? 1 Complex-valued neural networks fertilize electronics Thecomplex-valuedneuralnetworksarethenetworksthatdealwithcomplex- valued information by using complex-valued parameters and variables. They are extremely rich in variety. 1 In this chapter, we grasp the basic ideas lying in the complex-valued neural networks by glancing over an application exam- ple. Then we also obtain a bird’s-eye view of their present and prospective application fields so that we can enjoy the flavor before we go deep into the world of the complex-valued neural networks. 2 1.1 Imitate the brain, and surpass the brain Theartoftheartificialneuralnetworksisatechnologicalframeworkinwhich we introduce and/or imitate the functions, constructions, and dynamics of the brain to realize an adaptive and useful information processing. The brain is able to manage both the pattern processing problems and the symbolic processing ones. For example, when we find a correct traveling route in a complex transportation network such as metro network in a large city, we firstguessapossibleroutebyapatternprocessingand,afterward,weconfirm the details and sequential consistency. The principle and the mechanism of the brain functions are still unclear in total. However, the accumulation of physiological experiments has brought many important suggestions. Thebiologicalbrain,includingsensoryneuralnetworkssuchasretinaland cochlear networks, has various specific features. When it absorbs information 1 Various features and applications are found, for example, in a series of special sessions in international conferences such as [1], [2], [3], [4], [5], [6], [7] . 2 This Chapter is based on the article [8] (A.Hirose, Complex-valued neural net- works fertilize electronics, Journal of the IEICE, 87 (6):447–449, June 2004). Chapters in the multiple-author book [9] are also helpful to extend the first im- pression of the complex-valued neural networks. A. Hirose: Complex-valued neural networks fertilize electronics, Studies in Computational Intelligence(SCI)32,3–8(2006) www.springerlink.com (cid:1)c Springer-VerlagBerlinHeidelberg2006 4 1 Complex-valued neural networks fertilize electronics of events occurring in the world, it reconstructs the information according to their meaning for the person. It also preserves the relation among the infor- mation meanings. The brain’s manner to take in the information is roughly determinedbythenatureofthecells aswell astheconstructions ofthenerve networks. Additionally the brain changes itself being influenced by the in- formation presented by the environment. This change is the self-organization and / or learning in the neural system. The purpose of the self-organization and learning lies in the reconstruc- tionoftheinformationrepresentationinthebrainsothatthemancanutilize the information as effectively as possible. It is known that it is significantly important for the human neural network to adopt a representation suitable for the purpose assigned to respective network modules, in particular to a modulelocatedclosetohuman-environmentinterface.Therefore,thenetwork ofeachmodulealsopossessesaconstructionsuitableforprocessingrespective information specific to visual, auditory, or olfactory signals. The cerebral cor- tex has, however, a more homogeneous structure, i.e., the six-layer structure, so that one part of the cortex can be substituting for another part when an inability occurs. But it also self-organizes according to input signals. The modern electronics and communications provide us with a large va- riety of information. It is hence expected more and more in the engineering fields to develop new systems that process a wider range of information in more adaptive and effective manners just like we do, or better than we do. In other words, we need to build systems surpassing the brain by imitating the brain. Even in such cases, the effective self-organization and learning in- evitably require the information representations suitable for the purposes. 1.2 Create a “Superbrain” by enrichment of the information representation Let’sconsideranexample.Intheseyears,themeasurementtechnologyonthe basis of the interference of waves makes remarkable progress. Assume that we have a coherent lightwave transmitter and phase-sensitive eyes, i.e., an interferometric radar function, so that we can see the phase of the reflected lightwave [10] as shown in Fig.1.1. When a reflecting object approaches to us, the number of the wave (wave tops or bottoms) between the object and our eyes reduces. That is, the phase of the reflected lightwave progresses. Contrarily, when the object goes away, the phase is retarded. In this way, phase of the reflected lightwave expresses the distance between the object and us. The fluctuation is related to the unevenness and roughness of the object surface. Then, as we see the object coherently, our brain self-organizes in such a special manner that we can see the environment in a phase-sensitive way. That is, our brain looks the objective world adaptively on the basis of the amplitude and phase, i.e., “complex amplitude” or “phasor”, For example, 1.2 Createa“Superbrain”byenrichmentoftheinformationrepresentation 5 "The Superbrain" Phase-sensitive eyes Wavelength of electromagnetic wave Fig.1.1.Geographicalprofileacquisitionusingphase-sensitiveeyesandasuperbrain broughtupwiththespecialeyes.(FigureincludesthedatainDigitalMap50mGrid (Elevation), Geographical Survey Institute, with permission.) we are on a plane and see Mt. Fuji and Lake Yamanaka beneath the craft. If we have phase-sensitive eyes, our brain takes in the information of the height of Mt. Fuji, the shape and roughness of its skirt, statistical unevenness of distance determined by vegetation, and the fluctuation texture in distance. Then the brain recognizes the state of the ground surface with a pattern processing method in complex-amplitude information space. In Chapter 5, we present such an example of the phase-sensitive super- brain to see the region of Mt. Fuji and Lake Yamanaka. In Fig.5.2 on Page 91,youfindthedataofreflectionobservation,whileFig.5.6onPage96shows the recognition results. Figure 5.6(a) was generated by a conventional neural network that seesonly theintensity of thereflectedwave. Onthe other hand, Fig.5.6(b) is the result obtained by the phase-sensitive superbrain that sees the complex amplitude. The latter figure gives an impression completely dif- ferent from that of the former one. Mt. Fuji is segmented as a mountain cluster, which is a useful result for human beings to live in the world. We 6 1 Complex-valued neural networks fertilize electronics have also succeeded in visualization of antipersonnel plastic landmines with this complex-amplitude superbrain brought up with the phase-sensitive eyes (Chapter 6). One of the most important advantages of the complex-valued neural net- works(CVNNs)isgoodcompatibilitywithwavephenomena.TheCVNNsare suitable for the processes related to complex amplitude such as the interfer- ometric radar system mentioned above. In general, propagation and interfer- ence of electromagnetic wave are expressed by the magnitude of transmission and reflection, phase progression and retardation, superposition of fields, etc. These phenomena are expressed simply and naturally by the use of complex numbers. They are also related directly with the elemental processes in the CVNNssuchasweightingatsynapticconnections,i.e.,multiplicationsinam- plitude and shifts in phase, and summation of the weighted inputs. 1.3 Application fields expand rapidly and steadily Regarding research history of the CVNNs, we can trace back to a Paper [11] written by Aizenberg et al. in 1971. They presented not only the possibility of adaptive information processing on the basis of complex numbers, but also an analogy between the timing of neural firing and phase information, which may be useful to realize workable systems. That is, they considered a coding of phase information by using the progress or retardation of pulse timing. This idea is indeed suggestive even from the viewpoint of present research situations.Themostusefulapplicationsincludetheabove-mentionedcoherent electromagnetic system, where we pay attention to amplitude and phase of electromagneticwave.Insuchasystem,theamplitudecorrespondstoenergy, whereas the phase does to progress or retardation of time. The CVNNs deal withtheinformationdirectlyrelatedtosuchexistencethatformsthebasisof physical world. There are many other fields in which the CVNNs provide systems with appropriate information representations. Figure 1.2 is a diagram presenting application fields. Many are related to wave phenomena, e.g., active antennas in electromagnetism, communications and measurements using waves such as radar image processing, learning electron-wave devices, quantum computa- tion, ultrasonic imaging, analysis and synthesis in voice processing, and so on. The wavelength-dependent dynamics of optical circuit leads to adaptive optical routers in optical wavelength-division-multiplexed communications, variable optical connections, frequency-domain parallel information process- ing, etc. The carrier-frequency-dependent neural behavior realizes both the adaptability and controllability in neural networks. The compatibility of neural adaptability and controllability is closely re- lated to context-dependent behavior and emergence of volition in neural net- works and, hence, connected with so-called brainlike systems in the future. The periodicity in phase-information topology is applicable to systems that 1.3 Application fields expand rapidly and steadily 7 ceds escs Brainlike systems Learning / Self-organizing electronics, devices, New fields of Adaptive processing based on natural topology interfaces, sensors, and other subsystemschaos and fractalsand consistent relationship between signals / informationHighly Relative-Processing Complex- Higher- Stable 3-dimensionalActive antennas, Plastic optical Adaptive Signal proc. Learning Context-Quantum functional direction-based on domain order recurrent rotation in adaptive mobile connection, sensors and based on quantum dependent optical computing preserving complexity adaptive complex processing robotics and communications, optical routing, imaging devices /periodic information image devicesbehaviordevicesin amptlitude classificationprocessingand controlcolor processingradar systemsequalizationsystemsmetricsprocessingtransformand phase Compatibility of Memory Image Classification /Sonic / Ultra-Time-series Related physical existenElectromagnetic waveElectron waveLightwaveadaptability and processingand recallprocessingsegmentationsonic wave or applicatoin fielcontrollability Non-2-dimensionality Periodic Basic propertiChaos and fractals Good correspondence between information representation ControllableStable commutativity and direction-topology and dynamiin complex domainand physical existence / wave phenomenaadaptabilitydynamicsof quarternionpreservation in phase Coherent neural networks Quarternion, higher-order complex numbersfor processing waves Complex-valued neural networks Fig.1.2.Applicationfieldsofthecomplex-valuedneuralnetworks.Left-handside:Itemsclassifiedonthebasisofphysicalexistence.Right-handside:Itemsclassifiedonthebasisofthefeaturesofneuraldynamics.DetailsareexplainedinChapter3.3.(Reprintedfrom[8]:A.Hirose,“Complex-valuedneuralnetworksformorefertileelectronics,”JournaloftheIEICE,87(6):447–449,June2004,withpermission.) 8 1 Complex-valued neural networks fertilize electronics process information naturally having a cyclic structure, e.g., adaptive con- troller of traffic lights with periodic behavior. Such a research directly brings comfortandsafetytohumanbeings.Othertopicsincludenewdevelopmentin chaos and fractals, and use of quaternion and higher-order complex numbers. 1.4 Book organization In this book, we present and discuss basic ideas and fundamentals of CVNNs in Part I. First, in Chapter 2, we describe the backgrounds and reasons why theartoftheCVNNsbecomesimportantmoreandmore.Next,inChapter3, we present the features of CVNNs as well as in what applications they are especially useful. We also survey the history of CVNN researches. In Chap- ter 4, we explain the dynamics of processes, learning, and self-organization of CVNNs. We present dynamics of conventional (real-valued) neural networks first and, afterward, we extend them to those of CVNNs. Therefore, we ex- pect that even a beginner in conventional neural networks can easily absorb the basics of CVNNs. However, please consult the books listed in chapters, if needed, for further assistance required in understanding conventional neural networks. InPartII,wepresentseveralexamplesofapplicationsinCVNNs,proposed by the author’s research group, such as an interferometric radar imaging sys- temandanadaptivelightwaveinformationprocessingsystem.Wedescribethe features of self-organization and learning in these systems, and we show the effectiveness of the CVNNs that deals with phase information in waves. Theframeworkadoptedinthesystemsisnotonlyusefulinimagingandsens- ing using other wave phenomena such as sonic and ultrasonic waves, but also promising future development, e.g., in adaptive neural devices on the basis of electron wave [12],[9]. Furthermore, we explain in what manner a CVNN yields volition and developmental learning. We wish the ideas described in this book inspire the readers with new ideas more and more. 2 Neural networks: The characteristic viewpoints Before we describe complex-valued neural networks, first we review the basis of neural networks in general. The basic way of thinking is common to both the conventional (real-valued) and complex-valued neural networks. 2.1 Brain, artificial brain, artificial intelligence (AI), and neural networks What is the difference between computers we use in the daily life and neural networks in their basic ideas and constructions? To know the difference must be a useful compass to navigate around the world of complex-valued neural networks. The wonder of brain mechanism has been fascinating the human beings. Howdowerecognizeandcopewithenvironment?Whatarethemechanismsof therecognition,processing,learningandadaptation?Tobeginwith,whatare the self and consciousness? Such questions have attracted many people. Aris- totle in ancient Greek in the fourth century B.C. considered that the nature is made of four elements, i.e., fire, water, earth, and air, and they interchange withoneanotherinfluencedbythesun.Butlivingthingsadditionallypossess the soul, which is considered to bring us volition. That is, nutritive soul is given to plants, animals and humans, while perceptive soul is to animals and humans, and rational soul is only to humans. A little earlier than Aristotle, Chuang-tzuinancientChinaspeculatedonself.Inhisnarrative,hewasasleep onalakeside.Inhisdream,hewasabutterfly,andwasflyingaroundhappily. But he noticed that perhaps he might be actually a butterfly, and that he might be a human being only in the butterfly’s dream. What is the reality? What am I? In the late 1930s and 1940s, researchers attempted variously and com- prehensively to elucidate the excellent mechanisms of the human brain and utilize obtained knowledge to realize artificial brain. Various physiological measurementswereconductedsuchaselectroencephalographyandmembrane- A.Hirose:Neuralnetworks:Thecharacteristicviewpoints,StudiesinComputationalIntelligence (SCI)32,9–15(2006) www.springerlink.com (cid:1)c Springer-VerlagBerlinHeidelberg2006 10 2 Neural networks: The characteristic viewpoints potential recording using microelectrodes. The McCulloch & Pitts proposed a simplified neuron model. Norbert Wiener founded cybernetics. D. O. Hebb presented so-called Hebbian rule, a hypothesis on the learning mechanism at synapses[13].Furthermore,theconceptoftheTuringmachineandShannon’s information theory were also developed. Roughly speaking, the aim of these researches was the realization of artificial brain. However, after 1950s, the computers developed almost separately from the brain. The von Neumann computers, i.e., the present ordinary comput- ers, made a great progress [14]. Hardware progress improved the speed and capacity amazingly. The principal element of computer hardware was first vacuum tube, then transistor, integrated circuit (IC), large-scale integrated circuit (LSI), and now it is very-large-scale integrated circuit (VLSI). The development in scale and function enables us to deal with a large quantity of information bits very quickly. The computers have become hence useful to human being and are widely used now. In the von Neumann computers, the function is determined by software separately from physical existence. That is, software is directed to process symbolic information expressed by colorless bits on the basis of logic, i.e., hard rule. In this symbolic processing, the ex- pected process is clearly expressed by symbols and, therefore, the operation has no ambiguity. It has another advantage, i.e., the compatibility with re- ductionism. In other words, a problem to be solved may be reduced into a set of simpler problems, thanks to the clearness of logic. The modern society cannot go without computers even a single day. The artificial brain function realized by such modern computers has been called artificial intelligence (AI). In AI, the principal operation is symbolic processing on the basis of discrete mathematics. Provided that a problem is expressed logically clearly, then we can construct an efficient algorithm (processing procedure) to solve it. Many various and useful algorithms have been developed hitherto. If a problem is given only in an ambiguous way, we firstformalizetheproblemusingsymbolicrepresentation.Thenthecomputer searches an optimal action by using knowledge data and rules. An excellent example is the computer chess player “Deep Blue” that successfully beat the world chess champion in 1997. However, on the other hand, some problems have turned up. That is, most of real-world problems cannot be formalized clearly.Rulestobeusedareoftenvagueanduncertain.Thesearchspaceisalso too large for a computer to find an optimal action in a workable calculation time in most cases of realistically meaningful problems. Incidentally, in parallel with the development of von Neumann comput- ers, steady researches have been conducted to realize information processing more similar to that in the human brain, i.e., the neural networks. In these days, together with fuzzy processing and genetic algorithms, artificial neural networks is often called soft computing, or natural computing. As shown schematically in Fig.2.1, neural networks perform pattern processing, which is complementary to symbolic processing used in ordinary AI. The pattern processing deals with pattern information, i.e., information 2.1 Brain, artificial brain, artificial intelligence (AI), and neural networks 11 LEFT BRAIN RIGHT BRAIN Logical Intuitive Objective Subjective Analytic Synthetic Local + Sequential Global Elementalistic Holistic Centralized, Symbolic processing Distributed, Pattern processing Processing base: Bit Processing base: Metric (Artificially constructed information space) (Natural small-law-set infomation space) *Serial /time-sequential program *Continuous /holistic parallelism *Symbolic process i ng (AI) *Pattern processing Neural *Turing-machine modelled *Info space reflecting environment processing concept *Unconscious / mood processing von Neumann *Conscious processing Complementary ("Right br ain","Intuition","Sixth sense") Developped elementalistically, To develop synthetically, separate info, principle, existence close info, principle, existence Physical realization: Physical realization: Variable depending on purposes Specialized for bit processing *Linear/Nonlinear continuous processing *Bit memory, Logic circuit corresponding to environment *Bit-based transparent system *Unified processor, memory, interface, etc. *Time-sequential, procedural cirucuit *Parallel, synthetic circuit Fig. 2.1. Von Neumann information processing versus neural information process- ing. expressed like a picture, holistically all at once. For example, even a baby just after the birth follows moving things with his/her eyes. This action is an unconscious reaction required for him/her to survive in this world. Such non- symbolic and logic-independent processing is frequently observed in human beings on various levels, e.g., simple reaction, complex feeling, and even the sixth sense. On the other hand, most of adults perform also logical thinking such as calculation of change. Contrarily, when we process information time- sequentially, and the meaning and operation of the information are defined clearly, we call the treatment symbolic processing. Certainly, symbolic processing is a powerful method in some application fields. However, in 1990s, it has also been recognized that the symbolic- processing system is often inflexible and inadaptable, i.e., too hard.