PRINCIPLES OF ARTIFICIAL NEURAL NETWORKS 3rd Edition 8868hc_9789814522731_tp.indd 1 4/7/13 3:32 PM ADVANCED SERIES IN CIRCUITS AND SYSTEMS Editor-in-Charge: Wai-Kai Chen (Univ. Illinois, Chicago, USA) Associate Editor: Dieter A. Mlynski (Univ. Karlsruhe, Germany) Published Vol. 1: Interval Methods for Circuit Analysis by L. V. Kolev Vol. 2: Network Scattering Parameters by R. Mavaddat Vol. 3: Principles of Artificial Neural Networks by D Graupe Vol. 4: Computer-Aided Design of Communication Networks by Y-S Zhu and W K Chen Vol. 5: Feedback Networks: Theory and Circuit Applications by J Choma and W K Chen Vol. 6: Principles of Artificial Neural Networks, Second Edition by D Graupe Vol. 7: Principles of Artificial Neural Networks, Third Edition by D Graupe Advanced Series in Circuits and Systems – Vol. 7 PRINCIPLES OF ARTIFICIAL NEURAL NETWORKS 3rd Edition Daniel Graupe University of Illinois, Chicago, USA World Scientific NEW JERSEY • LONDON • SINGAPORE • BEIJING • SHANGHAI • HONG KONG • TAIPEI • CHENNAI 8868hc_9789814522731_tp.indd 2 4/7/13 3:32 PM Published by World Scientific Publishing Co. Pte. Ltd. 5 Toh Tuck Link, Singapore 596224 USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library. Advanced Series in Circuits and Systems — Vol. 7 PRINCIPLES OF ARTIFICIAL NEURAL NETWORKS Third Edition Copyright © 2013 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher. For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher. ISBN 978-981-4522-73-1 Printed in Singapore June25,2013 15:33 PrinciplesofArtificialNeuralNetworks(3rdEdn) ws-book975x65 Dedicated to the memory of my parents, to my wife Dalia, to our children, our daughters-in-law and our grandchildren It is also dedicated to the memory of Dr. Kate H. Kohn v June25,2013 15:33 PrinciplesofArtificialNeuralNetworks(3rdEdn) ws-book975x65 Acknowledgments I am most thankful to Hubert Kordylewski of the Department of Electrical Engineering and Computer Science of the University of Illinois at Chicago for his help towards the development of LAMSTAR network of Chapter 9 of this text. Hubert helped me with his advise in all 3 editions of this book. I am grateful to severalstudents who attended my classes on Neural Network at the Department of ElectricalEngineeringandComputerScienceoftheUniversityofIllinoisatChicago overthepastfourteenyearsandwhoallowedmetoappendprogramstheywroteas partofhomeworkassignmentsandcourseprojectsto variouschaptersofthisbook. TheyareVasanthArunachalam,AbdullaAl-Otaibi,GiovanniPaoloGibilisco,Sang Lee,MaximKolesnikov,HubertKordylewski,AlvinNg,EricNorth,MahaNujeimo, Michele Panzeri, Silvio Rizzi, Padmagandha Sahoo, Daniele Scarpazza, Sanjeeb Shah, Xiaoxiao Shi and Yunde Zhong. I am deeply indebted to the memory of Dr. Kate H. Kohn of Michael Reese Hospital, Chicago and of the College of Medicine of the University of Illinois at Chicago and to Dr. Boris Vern of the College of Medicine of the University of Illinois at Chicago for reviewing parts of the manuscript of this text and for their helpful comments. Ms. Barbara Aman and the production and editorial staff at World Scientific PublishingCompanyinSingaporewereextremelyhelpfulandpatientwithme dur- ing all phases of preparing this book for print. Last but not least, my sincere thanks to Steven Patt, my Editor at World Sci- entificPublishingCompany,throughoutalleditionsofthis book,forhiscontinuous help and support. vii June25,2013 15:33 PrinciplesofArtificialNeuralNetworks(3rdEdn) ws-book975x65 Preface to the Third Edition The Third Edition differs from the Second Edition in several important aspects. I added 4 new detailed case studies describing a variety of applications, (new Sections 6.D, 7.C, 9.C, 9.D) together with their respective source codes. This brings the total number of application to 19. This will allow the reader first-hand access to a wide range of different concrete APPLICATIONS of Neural Networks ranging from medicine to constellation detection, thus establishing the main claim of Neural Networks, namely the claim of its GENERALITY. Thenew casestudiesinclude anapplicationtoanon-linearpredictionproblems (case study 9.C), which are indeed a field where artificial neural networks are and will play a major role. This case study also compares performances of two differ- ent neural networks in terms of accuracy and computational time, for the specific problem of that case study. Also, two new Section (9.6 and 9.8) were added to Chapter 9. Text organization is also modified. The Chapter on the Large Memory Storage and Retrieval Neural Network (LAMSTAR) is moved from Chapter 13 of the Sec- ond Edition, to become Chapter 9 in the present Edition. Consequently, the old Chapter 9 to 12 are now Chapters 10 to 13. This allows teaching and self-study to follow the main Artificial Neural Networks (ANN) in a more logical order in terms of basic principles and generality. We consider that in short courses, Chapters 1 to 9 will thus become the core of a course on ANN. It is hoped that these text and this enhanced Edition can serve to show and to persuade scientists, engineers and program developers in areas ranging from medicine to finance and beyond, of the value and the power of ANN in problems that are ill-defined, highly non-linear, stochastic and of time-varying dynamics and which often appear to be beyond solution. Additional corrections and minor modifications are also included, as are other updates based on recent developments including those relating to the author’s research. Daniel Graupe Chicago, IL March 2013 ix