Complex-Valued Neural Networks IEEE Press 445 Hoes Lane Piscataway, NJ 08854 IEEE Press Editorial Board 2013 John Anderson, Editor in Chief Linda Shafer Saeid Nahavandi George Zobrist George W. Arnold David Jacobson Tariq Samad Ekram Hossain Mary Lanzerotti Dmitry Goldgof Om P. Malik Kenneth Moore, Director of IEEE Book and Information Services (BIS) Technical Reviewers George M. Georgiou Gouhei Tanaka Complex-Valued Neural Networks Advances and Applications Edited by Akira Hirose The University of Tokyo Φ IEEE Press Series on Computational Intelligence David B. Fogel, Series Editor 4 IEEE IEEE PRESS WILEY Copyright © 2013 by The Institue of Electrical and Electronics Engineers. All rights reserved. Published by John Wiley & Sons, Inc., Hoboken, New Jersey. Published simultaneously in Canada. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permission. Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representation or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. For general information on our other products and services please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print, however, may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com. Library of Congress Cataloging-in-Publication Data is available. ISBN 9781118344606 Printed in the United States of America. 10 9 8 7 6 5 4 3 21 CONTRIBUTORS CHAPTER 1 AKIRA HIROSE, The University of Tokyo, Tokyo, Japan CHAPTER 2 SIMONE FIORI, Universita Politecnica delle Marche, Ancona, Italy CHAPTER 3 TOHRU NiTTA, National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Japan CHAPTER 4 MD. FAIJUL AMIN, University of Fukui, Fukui, Japan; Khulna University of Engi- neering and Technology, Khulna, Bangladesh KAZUYUKI MURASE, University of Fukui, Fukui, Japan v VI CONTRIBUTORS CHAPTER 5 TEIJIRO ISOKAWA, University of Hyogo, Hyogo, Japan HARUHIKO NISHIMURA, University of Hyogo, Hyogo, Japan NOBUYUKI MATSUI, University of Hyogo, Hyogo, Japan CHAPTER 6 YASUAKI KUROE, Kyoto Institute of Technology, Kyoto, Japan CHAPTER 7 RAMASAMY SAVITHA, Nanyang Technological University, Singapore SUNDARAM SURESH, Nanyang Technological University, Singapore NARASIMHAN SUNDARARAJAN, Sri Jaya Chamarajendra College of Engineering (SJCE), Mysore, India CHAPTER 8 NIKOLAY MANYAKOV, KU Leuven, Leuven, Belgium IGOR AIZENBERG, Texas A&M University-Texarkana, Texarkana, Texas, U.S.A. NILOLAY CHUMERIN, KU Leuven, Leuven, Belgium MARK M. VAN HULLE, KU Leuven, Leuven, Belgium CHAPTER 9 XiA HONG, University of Reading, Reading, U.K. SHENG CHEN, University of Southampton, Southampton, U.K.; King Abdulaziz University, Jeddah, Saudi Arabia CHRIS J. HARRIS, University of Southampton, Southampton, U.K. CHAPTER 10 WAI KIT WONG, Multimedia University, Melaka, Malaysia GIN CHONG LEE, Multimedia University, Melaka, Malaysia CHU KIONG Loo, University of Malaya, Kuala Lumpur, Malaysia WAY SOONG LIM, Multimedia University, Melaka, Malaysia RAYMOND LOCK, Multimedia University, Melaka, Malaysia CONTENTS Preface xv 1 Application Fields and Fundamental Merits 1 Akira Hirose 1.1 Introduction 1 1.2 Applications of Complex-Valued Neural Networks 2 1.2.1 Antenna Design 2 1.2.2 Estimation of Direction of Arrival and Beamforming 3 1.2.3 Radar Imaging 3 1.2.4 Acoustic Signal Processing and Ultrasonic Imaging 3 1.2.5 Communications Signal Processing 3 1.2.6 Image Processing 4 1.2.7 Social Systems Such as Traffic and Power Systems 4 1.2.8 Quantum Devices Such as Superconductive Devices 4 1.2.9 Optical/Lightwave Information Processing Including Carrier-Frequency Multiplexing 4 vii viii CONTENTS 1.2.10 Hypercomplex-Valued Neural Networks 5 1.3 What is a complex number? 5 1.3.1 Geometric and Intuitive Definition 5 1.3.2 Definition as Ordered Pair of Real Numbers 6 1.3.3 Real 2x2 Matrix Representation 7 1.4 Complex numbers in feedforward neural networks 8 1.4.1 Synapse and Network Function in Layered Feedforward Neural Networks 9 1.4.2 Circularity 11 1.5 Metric in complex domain 12 1.5.1 Metric in Complex-Valued Self-Organizing Map 12 1.5.2 Euclidean Metric 12 1.5.3 Complex Inner-Product Metric 14 1.5.4 Comparison Between Complex Inner Product and Euclidean Distance 14 1.5.5 Metric in Correlation Learning 15 1.6 Experiments to elucidate the generalization characteristics 16 1.6.1 Forward Processing and Learning Dynamics 17 1.6.2 Experimental Setup 21 1.6.3 Results 24 1.7 Conclusions 26 References 27 2 Neural System Learning on Complex-Valued Manifolds 33 Simone Fiori 2.1 Introduction 34 2.2 Learning Averages over the Lie Group of Unitary Matrices 35 2.2.1 Differential-Geometric Setting 36 2.2.2 An Averaging Procedure over the Lie Group of Unitary Matrices 37 2.3 Riemannian-Gradient-Based Learning on the Complex Matrix-Hypersphere 41 2.3.1 Geometric Characterization of the Matrix Hypersphere 43 2.3.2 Geodesic-Stepping Optimization Method 45 2.3.3 Application to Optimal Precoding in MIMO Broadcast Channels 46 2.4 Complex ICA Applied to Telecommunications 49 CONTENTS IX 2.4.1 Complex-Weighted Rigid-Body Learning Equations for ICA 51 2.4.2 Application to the Blind Separation of QAM/PSK Signals 53 2.5 Conclusion 53 References 55 3 TV-Dimensional Vector Neuron and Its Application to the TV-Bit Parity Problem 59 Tohru Nitta 3.1 Introduction 59 3.2 Neuron Models with High-Dimensional Parameters 60 3.2.1 Complex-Valued Neuron 60 3.2.2 Hyperbolic Neuron 61 3.2.3 Three-Dimensional Vector Neuron 61 3.2.4 Three-Dimensional Vector Product Neuron 62 3.2.5 Quaternary Neuron 63 3.2.6 Clifford Neuron 63 3.3 iV-Dimensional Vector Neuron 65 3.3.1 iV-Dimensional Vector Neuron Model 65 3.3.2 Decision Boundary 65 3.3.3 iV-Bit Parity Problem 67 3.3.4 A Solution 67 3.4 Discussion 69 3.5 Conclusion 70 References 71 4 Learning Algorithms in Complex-Valued Neural Networks using Wirtinger Calculus 75 Md. Faijul Amin and Kazuyuki Murase 4.1 Introduction 76 4.2 Derivatives in Wirtinger Calculus 78 4.3 Complex Gradient 80 4.4 Learning Algorithms for Feedforward CVNNs 82 4.4.1 Complex Gradient Descent Algorithm 82 4.4.2 Complex Levenberg-Marquardt Algorithm 86 4.4.3 Computer Simulations 89 4.5 Learning Algorithms for Recurrent CVNNs 91 X CONTENTS 4.5.1 Complex Real-Time Recurrent Learning Algorithm 92 4.5.2 Complex Extended Kaiman Filter Algorithm 95 4.5.3 Computer Simulations 96 4.6 Conclusion 99 References 100 5 Quaternionic Neural Networks for Associative Memories 103 Teijiro Isokawa, Haruhiko Nishimura, and Nobuyuki Matsui 5.1 Introduction 104 5.2 Quaternionic Algebra 105 5.2.1 Definition of Quaternion 105 5.2.2 Phase Representation of Quaternion 106 5.2.3 Quaternionic Analyticity 106 5.3 Stability of Quaternionic Neural Networks 108 5.3.1 Network with Bipolar State Neurons 108 5.3.2 Network with Continuous State Neurons 111 5.3.3 Network with Continuous State Neurons Having Local Analytic Activation Function 115 5.3.4 Network with Multistate Neurons 118 5.4 Learning Schemes for Embedding Patterns 124 5.4.1 Hebbian Rule 124 5.4.2 Projection Rule 125 5.4.3 Iterative Learning for Quaternionic Multistate Neural Network 126 5.5 Conclusion 128 References 129 6 Models of Recurrent Clifford Neural Networks and Their Dynamics 133 Yasuaki Kuroe 6.1 Introduction 134 6.2 Clifford Algebra 134 6.2.1 Definition 134 6.2.2 Basic Properties and Algebraic Basis 136 6.3 Hopfield-Type Neural Networks and Their Energy Functions 137 6.4 Models of Hopfield-Type Clifford Neural Networks 139 6.5 Definition of Energy Functions 140