PRINCIPLES OF ARTIFICIAL NEURAL NETWORKS 2nd Edition ADVANCED SERIES IN CIRCUITS AND SYSTEMS Editor-in-Charge: Wai-Kai Chen (Univ. Illinois, Chicago, USA) Associate Editor: Dieter A. Mlynski (Univ. Karlsruhe, Germany) Published Vol. 1: Interval Methods for Circuit Analysis by L. V. Kolev Vol. 2: Network Scattering Parameters by R. Mavaddat Vol. 3: Principles of Artificial Neural Networks by D Graupe Vol. 4: Computer-Aided Design of Communication Networks by Y-S Zhu & W K Chen Vol. 5: Feedback Networks: Theory & Circuit Applications by J Choma & W K Chen Vol. 6: Principles of Artificial Neural Networks (2nd Edition) by D Graupe Steven - Principles of Arti Neural.pmd 2 1/30/2007, 4:11 PM Advanced Series on Circuits and Systems – Vol. 6 PRINCIPLES OF ARTIFICIAL NEURAL NETWORKS 2nd Edition Daniel Graupe University of lllinois, Chicago, USA World Scientific NEW JWRSEY . LONDON . SINGAPORE . BEIJING . SHANGHAI . HONG KONG . TAIPEI . CHENNAI Published by World Scientific Publishing Co. Pte. Ltd. 5 Toh Tuck Link, Singapore 596224 USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library. PRINCIPLES OF ARTIFICIAL NEURAL NETWORKS (2nd Edition) Advanced Series on Circuits and Systems – Vol. 6 Copyright © 2007 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher. For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher. ISBN-13 978-981-270-624-9 ISBN-10 981-270-624-0 Printed in Singapore. Steven - Principles of Arti Neural.pmd 1 1/30/2007, 4:11 PM January30,2007 16:24 WorldScienti(cid:12)cBook-9.75inx6.5in ws-book975x65 Dedicated to the memory of my parents, to my wife Dalia, to our children, our daughters-in-law and our grandchildren It is also dedicated to the memory of Dr. Kate H Kohn v TThhiiss ppaaggee iinntteennttiioonnaallllyy lleefftt bbllaannkk January30,2007 16:24 WorldScienti(cid:12)cBook-9.75inx6.5in ws-book975x65 Acknowledgments I am most thankful to Hubert Kordylewski of the Department of Electrical Engineering and Computer Science of the University of Illinois at Chicago for his help towards the development of LAMSTAR network of Chapter 13 of this text. I am grateful to several students who attended my classes on Neural Network at the Department of Electrical Engineering and Computer Science of the University of Illinois at Chicago over the past fourteen years and who allowed me to append programs they wrote as part of homework assignments and course projects to var- ious chapters of this book. They are Vasanth Arunachalam, Sang Lee, Maxim Kolesnikov, Hubert Kordylewski, Maha Nujeimo, Michele Panzeri, Padmagandha Sahoo, Daniele Scarpazza, Sanjeeb Shah and Yunde Zhong. I am deeply indebted to the memory of Dr. Kate H. Kohn of Michael Reese Hospital, Chicago and of the College of Medicine of the University of Illinois at Chicago and to Dr. Boris Vern of the College of Medicine of the University ofIllinoisatChicagoforreviewingpartsofthemanuscriptofthistextandfortheir helpful comments. Ms. Barbara Aman and the production and editorial sta(cid:11) at World Scienti(cid:12)c Publishing Company in Singapore were extremely helpful and patient with me during all phases of preparing this book for print. vii TThhiiss ppaaggee iinntteennttiioonnaallllyy lleefftt bbllaannkk January30,2007 16:24 WorldScienti(cid:12)cBook-9.75inx6.5in ws-book975x65 Preface to the First Edition Thisbookevolvedfromthelecturenotesofa(cid:12)rst-yeargraduatecourseentitled \Neural Networks" which I taught at the Department of Electrical Engineering and Computer Science of the University of Illinois at Chicagoover the years1990{ 1996. Whereas that course was a (cid:12)rst-year graduate course, several Senior-Year undergraduate students from di(cid:11)erent engineering departments, attended it with little di(cid:14)culty. It was mainly for historical and scheduling reasons that the course wasagraduatecourse,sincenosuchcourseexistedinourprogramofstudiesandin thecurriculaofmostU.S.universitiesintheSeniorYearUndergraduateprogram. I thereforeconsiderthisbook,whichcloselyfollowstheselecturenotes,tobesuitable for such undergraduate students. Furthermore, it should be applicable to students at that level from essentially every science and engineering University department. Itsprerequisitesarethemathematicalfundamentalsintermsofsomelinearalgebra and calculus, and computational programming skills (not limited to a particular programminglanguage) that all such students possess. Indeed, I strongly believe that Neural Networks are a (cid:12)eld of both intellectual interest and practical value to all such students and young professionals. Arti(cid:12)cial neural networks not only provide an understanding into an important computa- tional architectureand methodology,but they also provideanunderstanding (very simpli(cid:12)ed, of course) of the mechanism of the biologicalneural network. Neural networks were until recently considered as a \toy" by many computer engineers and business executives. This was probably somewhat justi(cid:12)ed in the past, since neural nets could at best apply to small memories that were analyzable just as successfully by other computational tools. I believe (and I tried in the laterchaptersbelowtogivesomedemonstrationtosupportthis belief) that neural networksareindeed avalid, and presently, the only e(cid:14)cient tool, to deal with very large memories. Thebeautyofsuchnetsisthattheycanallowandwillinthenear-futureallow, for instance, a computer user to overcome slight errors in representation, in pro- gramming (missing a trivial but essential command such as a period or any other symbol or character) and yet have the computer execute the command. This will obviously requirea neural network bu(cid:11)er between the keyboardand the main pro- ix