Computational Intelligence Paradigms Theory and Applications using MATLAB® © 2010 by Taylor and Francis Group, LLC Computational Intelligence Paradigms Theory and Applications using MATLAB® S. sumathi surekha p. Boca Raton London New York CRC Press is an imprint of the Taylor & Francis Group, an informa business A CHAPMAN & HALL BOOK © 2010 by Taylor and Francis Group, LLC MATLAB ® and Simulink ® are trademarks of the Math Works, Inc. and are used with permission. The Mathworks does not warrant the accuracy of the text or exercises in this book. This book’s use or discussion of MATLAB ® and Simulink ® software or related products does not constitute endorsement or sponsorship by the Math Works of a particular peda- gogical approach or particular use of the MATLAB ® and Simulink ® software. CRC Press Taylor & Francis Group 6000 Broken Sound Parkway NW, Suite 300 Boca Raton, FL 33487-2742 © 2010 by Taylor and Francis Group, LLC CRC Press is an imprint of Taylor & Francis Group, an Informa business No claim to original U.S. Government works Printed in the United States of America on acid-free paper 10 9 8 7 6 5 4 3 2 1 International Standard Book Number: 978-1-4398-0902-0 (Hardback) This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the valid- ity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint. Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or uti- lized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopy- ing, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers. For permission to photocopy or use material electronically from this work, please access www.copyright.com (http:// www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. Library of Congress Cataloging‑in‑Publication Data Sumathi, S., 1968- Computational intelligence paradigms : theory & applications using MATLAB / S. Sumathi and Surekha Paneerselvam. p. cm. Includes bibliographical references and index. ISBN 978-1-4398-0902-0 (hard back : alk. paper) 1. Computational intelligence. 2. MATLAB. I. Paneerselvam, Surekha, 1980- II. Title. Q342.S94 2010 006.3--dc22 2009022113 Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com and the CRC Press Web site at http://www.crcpress.com © 2010 by Taylor and Francis Group, LLC Contents Preface xvii 1 Computational Intelligence 1 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Primary Classes of Problems for CI Techniques . . . . 5 1.2.1 Optimization . . . . . . . . . . . . . . . . . . . 5 1.2.2 NP-Complete Problems . . . . . . . . . . . . . 6 1.3 Neural Networks . . . . . . . . . . . . . . . . . . . . . 7 1.3.1 Feed Forward Neural Networks . . . . . . . . . 8 1.4 Fuzzy Systems . . . . . . . . . . . . . . . . . . . . . . 9 1.4.1 Fuzzy Sets . . . . . . . . . . . . . . . . . . . . 9 1.4.2 Fuzzy Controllers . . . . . . . . . . . . . . . . 11 1.5 Evolutionary Computing . . . . . . . . . . . . . . . . . 12 1.5.1 Genetic Algorithms . . . . . . . . . . . . . . . 13 1.5.2 Genetic Programming . . . . . . . . . . . . . . 14 1.5.3 Evolutionary Programming . . . . . . . . . . . 15 1.5.4 Evolutionary Strategies . . . . . . . . . . . . . 15 1.6 Swarm Intelligence . . . . . . . . . . . . . . . . . . . . 16 1.7 Other Paradigms . . . . . . . . . . . . . . . . . . . . . 17 1.7.1 Granular Computing . . . . . . . . . . . . . . 18 1.7.2 Chaos Theory . . . . . . . . . . . . . . . . . . 20 1.7.3 Artificial Immune Systems . . . . . . . . . . . 21 1.8 Hybrid Approaches . . . . . . . . . . . . . . . . . . . . 22 1.9 Relationship with Other Paradigms . . . . . . . . . . . 23 1.10 Challenges To Computational Intelligence . . . . . . . 25 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . 27 2 Artificial Neural Networks with MATLAB 29 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 29 2.2 A Brief History of Neural Networks . . . . . . . . . . . 29 2.3 Artificial Neural Networks . . . . . . . . . . . . . . . . 32 2.3.1 Comparison of Neural Network to the Brain . 33 2.3.2 Artificial Neurons . . . . . . . . . . . . . . . . 33 © 2010 by Taylor and Francis Group, LLC v vi 2.3.3 Implementation of Artificial Neuron Electroni- cally . . . . . . . . . . . . . . . . . . . . . . . . 35 2.3.4 Operations of Artificial Neural Network . . . . 37 2.3.5 Training an Artificial Neural Network . . . . . 40 2.3.6 Comparison between Neural Networks, Tradi- tional Computing, and Expert Systems . . . . 44 2.4 Neural Network Components . . . . . . . . . . . . . . 46 2.4.1 Teaching an Artificial Neural Network . . . . . 52 2.4.2 Learning Rates . . . . . . . . . . . . . . . . . . 54 2.4.3 Learning Laws . . . . . . . . . . . . . . . . . . 55 2.4.4 MATLAB Implementation of Learning Rules . 61 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . 67 3 Artificial Neural Networks - Architectures and Algo- rithms 69 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 69 3.2 LayeredArchitectures . . . . . . . . . . . . . . . . . . 71 3.2.1 Single-Layer Networks . . . . . . . . . . . . . . 71 3.2.2 Multilayer Networks . . . . . . . . . . . . . . . 73 3.3 Prediction Networks . . . . . . . . . . . . . . . . . . . 75 3.3.1 The Perceptron . . . . . . . . . . . . . . . . . 75 3.3.2 MATLABImplementationofaPerceptronNet- work . . . . . . . . . . . . . . . . . . . . . . . 79 3.3.3 Feedforward Back-PropagationNetwork . . . . 83 3.3.4 Implementation of BPN Using MATLAB . . . 91 3.3.5 Delta Bar Delta Network . . . . . . . . . . . . 95 3.3.6 Extended Delta Bar Delta . . . . . . . . . . . 97 3.3.7 Directed Random Search Network . . . . . . . 100 3.3.8 Functional Link Artificial Neural Network (FLANN) or Higher-Order Neural Network . . 102 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . 106 4 Classification and Association Neural Networks 109 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 109 4.2 Neural Networks Based on Classification . . . . . . . . 109 4.2.1 Learning Vector Quantization . . . . . . . . . 110 4.2.2 Implementation of LVQ in MATLAB . . . . . 115 4.2.3 Counter-PropagationNetwork . . . . . . . . . 116 4.2.4 Probabilistic Neural Network . . . . . . . . . . 123 4.2.5 Implementation of the Probabilistic Neural Net Using MATLAB . . . . . . . . . . . . . . . . . 128 © 2010 by Taylor and Francis Group, LLC vii 4.3 Data Association Networks . . . . . . . . . . . . . . . 129 4.3.1 Hopfield Network . . . . . . . . . . . . . . . . 130 4.3.2 Implementation of Hopfield Network in MAT- LAB. . . . . . . . . . . . . . . . . . . . . . . . 133 4.3.3 Boltzmann Machine . . . . . . . . . . . . . . . 134 4.3.4 Hamming Network . . . . . . . . . . . . . . . . 137 4.3.5 Bi-Directional Associative Memory. . . . . . . 141 4.4 Data Conceptualization Networks . . . . . . . . . . . . 146 4.4.1 Adaptive Resonance Network . . . . . . . . . . 146 4.4.2 Implementation of ART Algorithm in MATLAB 151 4.4.3 Self-Organizing Map . . . . . . . . . . . . . . . 153 4.5 Applications Areas of ANN . . . . . . . . . . . . . . . 158 4.5.1 Language Processing . . . . . . . . . . . . . . 159 4.5.2 Character Recognition. . . . . . . . . . . . . . 160 4.5.3 Data Compression . . . . . . . . . . . . . . . . 160 4.5.4 Pattern Recognition . . . . . . . . . . . . . . . 160 4.5.5 Signal Processing . . . . . . . . . . . . . . . . 161 4.5.6 Financial . . . . . . . . . . . . . . . . . . . . . 161 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . 162 5 MATLAB Programs to Implement Neural Networks 165 5.1 Illustration 1: Coin Detection Using Euclidean Distance (Hamming Net) . . . . . . . . . . . . . . . . . . . . . . 165 5.2 Illustration2:LearningVectorQuantization-Clustering Data Drawn from Different Regions . . . . . . . . . . 171 5.3 Illustration 3: Character Recognition Using Kohonen Som Network . . . . . . . . . . . . . . . . . . . . . . . 174 5.4 Illustration 4: The Hopfield Network as an Associative Memory . . . . . . . . . . . . . . . . . . . . . . . . . . 182 5.5 Illustration 5: Generalized Delta Learning Rule and Back-Propagationof Errors for a Multilayer Network . 187 5.6 Illustration 6: Classification of Heart Disease Using Learning Vector Quantization . . . . . . . . . . . . . . 189 5.7 Illustration7: Neural Network Using MATLAB Simulink 198 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . 200 6 MATLAB-Based Fuzzy Systems 203 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 203 6.2 Imprecision and Uncertainty . . . . . . . . . . . . . . 205 6.3 Crisp and Fuzzy Logic . . . . . . . . . . . . . . . . . . 205 6.4 Fuzzy Sets . . . . . . . . . . . . . . . . . . . . . . . . . 207 © 2010 by Taylor and Francis Group, LLC viii 6.5 Universe . . . . . . . . . . . . . . . . . . . . . . . . . . 209 6.6 Membership Functions . . . . . . . . . . . . . . . . . . 210 6.6.1 Types of Membership Functions . . . . . . . . 212 6.6.2 Membership Functions in the MATLAB Fuzzy Logic Toolbox . . . . . . . . . . . . . . . . . . 214 6.6.3 MATLAB Code to Simulate MembershipFunc- tions . . . . . . . . . . . . . . . . . . . . . . . 217 6.6.4 TranslationofParametersbetweenMembership Functions Using MATLAB . . . . . . . . . . . 223 6.7 Singletons . . . . . . . . . . . . . . . . . . . . . . . . . 224 6.8 Linguistic Variables . . . . . . . . . . . . . . . . . . . 225 6.9 Operations on Fuzzy Sets . . . . . . . . . . . . . . . . 225 6.9.1 Fuzzy Complements . . . . . . . . . . . . . . . 227 6.9.2 Fuzzy Intersections: t-norms . . . . . . . . . . 230 6.9.3 Fuzzy Unions: t-conorms . . . . . . . . . . . . 233 6.9.4 Combinations of Operations . . . . . . . . . . 235 6.9.5 MATLAB Codes for Implementation of Fuzzy Operations . . . . . . . . . . . . . . . . . . . . 236 6.9.6 AggregationOperations . . . . . . . . . . . . . 239 6.10 Fuzzy Arithmetic . . . . . . . . . . . . . . . . . . . . . 242 6.10.1 Arithmetic Operations on Intervals . . . . . . 243 6.10.2 Arithmetic Operations on Fuzzy Numbers . . 244 6.10.3 Fuzzy Arithmetic Using MATLAB Fuzzy Logic Toolbox . . . . . . . . . . . . . . . . . . . . . . 246 6.11 Fuzzy Relations . . . . . . . . . . . . . . . . . . . . . . 247 6.12 Fuzzy Composition . . . . . . . . . . . . . . . . . . . . 251 6.12.1 MATLAB Code to Implement Fuzzy Composi- tion . . . . . . . . . . . . . . . . . . . . . . . . 254 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . 260 7 Fuzzy Inference and Expert Systems 261 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 261 7.2 Fuzzy Rules . . . . . . . . . . . . . . . . . . . . . . . . 261 7.2.1 Generation of Fuzzy Rules . . . . . . . . . . . 262 7.2.2 Disintegration of Rules . . . . . . . . . . . . . 262 7.2.3 Aggregationof Rules . . . . . . . . . . . . . . 263 7.3 Fuzzy Expert System Model . . . . . . . . . . . . . . . 264 7.3.1 Fuzzification . . . . . . . . . . . . . . . . . . . 264 7.3.2 Fuzzy Rule Base and Fuzzy IF-THEN Rules . 265 7.3.3 Fuzzy Inference Machine . . . . . . . . . . . . 266 7.3.4 Defuzzification . . . . . . . . . . . . . . . . . . 267 © 2010 by Taylor and Francis Group, LLC ix 7.3.5 Implementation of Defuzzification using MAT- LAB Fuzzy Logic Toolbox . . . . . . . . . . . 273 7.4 Fuzzy Inference Methods . . . . . . . . . . . . . . . . 276 7.4.1 Mamdani’s Fuzzy Inference Method . . . . . . 278 7.4.2 Takagi–Sugeno Fuzzy Inference Method . . . . 281 7.4.3 Tsukamoto Fuzzy Inference Method . . . . . . 282 7.5 Fuzzy Inference Systems in MATLAB . . . . . . . . . 283 7.5.1 Mamdani-Type Fuzzy Inference . . . . . . . . 286 7.5.2 Sugeno-Type Fuzzy Inference . . . . . . . . . 292 7.5.3 Conversionof Mamdani to Sugeno System . . 295 7.6 Fuzzy Automata and Languages . . . . . . . . . . . . 297 7.7 Fuzzy Control . . . . . . . . . . . . . . . . . . . . . . . 298 7.7.1 Fuzzy Controllers . . . . . . . . . . . . . . . . 300 7.7.2 A Fuzzy Controller in MATLAB . . . . . . . . 302 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . 305 8 MATLAB Illustrations on Fuzzy Systems 307 8.1 Illustration 1: Application of Fuzzy Controller Using MATLAB — Fuzzy Washing Machine . . . . . . . . . 307 8.2 Illustration 2 - Fuzzy Control System for a Tanker Ship 317 8.3 Illustration 3 - Approximation of Any Function Using Fuzzy Logic . . . . . . . . . . . . . . . . . . . . . . . . 336 8.4 Illustration 4 - Building Fuzzy Simulink Models . . . . 343 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . 348 9 Neuro-Fuzzy Modeling Using MATLAB 351 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 351 9.2 Cooperative and Concurrent Neuro-Fuzzy Systems . . 352 9.3 Fused Neuro-Fuzzy Systems . . . . . . . . . . . . . . . 352 9.3.1 Fuzzy Adaptive Learning Control Network (FALCON) . . . . . . . . . . . . . . . . . . . . 353 9.3.2 Adaptive Neuro-Fuzzy Inference System (AN- FIS) . . . . . . . . . . . . . . . . . . . . . . . . 353 9.3.3 Generalized Approximate Reasoning-Based In- telligent Control (GARIC) . . . . . . . . . . . 355 9.3.4 Neuro-Fuzzy Control (NEFCON) . . . . . . . 356 9.3.5 Fuzzy Inference and Neural Network in Fuzzy Inference Software (FINEST) . . . . . . . . . . 360 9.3.6 Fuzzy Net (FUN) . . . . . . . . . . . . . . . . 362 9.3.7 Evolving Fuzzy Neural Network (EFuNN) . . . 363 © 2010 by Taylor and Francis Group, LLC x 9.3.8 Self–Constructing Neural Fuzzy Inference Net- work (SONFIN) . . . . . . . . . . . . . . . . . 364 9.3.9 Evolutionary Design of Neuro-Fuzzy Systems . 364 9.4 Hybrid Neuro-Fuzzy Model — ANFIS . . . . . . . . . 367 9.4.1 ArchitectureofAdaptiveNeuro-FuzzyInference System . . . . . . . . . . . . . . . . . . . . . . 367 9.4.2 Hybrid Learning Algorithm . . . . . . . . . . 370 9.5 Classification and Regression Trees . . . . . . . . . . . 372 9.5.1 CART — Introduction . . . . . . . . . . . . . 372 9.5.2 Node Splitting Criteria . . . . . . . . . . . . . 373 9.5.3 Classification Trees . . . . . . . . . . . . . . . 374 9.5.4 Regression Trees . . . . . . . . . . . . . . . . . 375 9.5.5 Computational Issues of CART. . . . . . . . . 375 9.5.6 Computational Steps . . . . . . . . . . . . . . 376 9.5.7 Accuracy Estimation in CART . . . . . . . . . 379 9.5.8 Advantages of Classification and Regression Trees . . . . . . . . . . . . . . . . . . . . . . . 380 9.6 Data Clustering Algorithms . . . . . . . . . . . . . . . 382 9.6.1 System Identification Using Fuzzy Clustering . 382 9.6.2 Hard C-Means Clustering . . . . . . . . . . . . 383 9.6.3 Fuzzy C-Means (FCM) Clustering . . . . . . . 386 9.6.4 Subtractive Clustering. . . . . . . . . . . . . . 388 9.6.5 Experiments . . . . . . . . . . . . . . . . . . . 389 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . 393 10 Neuro-Fuzzy Modeling Using MATLAB 395 10.1 Illustration 1 - Fuzzy Art Map . . . . . . . . . . . . . 395 10.2 Illustration 2: Fuzzy C-Means Clustering — Compara- tive Case Study . . . . . . . . . . . . . . . . . . . . . . 402 10.3 Illustration 3 - Kmeans Clustering . . . . . . . . . . . 403 10.4 Illustration 4 - Neuro-Fuzzy System Using Simulink . 411 10.5 Illustration 5 - Neuro-Fuzzy System Using Takagi– Sugeno and ANFIS GUI of MATLAB . . . . . . . . . 414 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . 418 11 Evolutionary Computation Paradigms 419 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 419 11.2 Evolutionary Computation . . . . . . . . . . . . . . . 420 11.3 Brief History of Evolutionary Computation . . . . . . 422 11.4 Biologicaland Artificial Evolution . . . . . . . . . . . 423 11.4.1 Expressions Used in Evolutionary Computation 423 © 2010 by Taylor and Francis Group, LLC xi 11.4.2 Biological Evolution Inspired by Nature . . . . 423 11.4.3 Evolutionary Biology . . . . . . . . . . . . . . 425 11.5 Flow Diagram of a Typical Evolutionary Algorithm . 428 11.6 Models of Evolutionary Computation . . . . . . . . . . 430 11.6.1 Genetic Algorithms (GA) . . . . . . . . . . . . 430 11.6.2 Genetic Programming (GP). . . . . . . . . . . 431 11.6.3 Evolutionary Programming (EP) . . . . . . . . 435 11.6.4 Evolutionary Strategies (ESs) . . . . . . . . . 436 11.7 Evolutionary Algorithms . . . . . . . . . . . . . . . . . 439 11.7.1 Evolutionary Algorithms Parameters . . . . . 442 11.7.2 Solution Representation . . . . . . . . . . . . . 442 11.7.3 Fitness Function . . . . . . . . . . . . . . . . . 443 11.7.4 Initialization of Population Size . . . . . . . . 444 11.7.5 Selection Mechanisms . . . . . . . . . . . . . . 444 11.7.6 CrossoverTechnique . . . . . . . . . . . . . . . 451 11.7.7 Mutation Operator . . . . . . . . . . . . . . . 455 11.7.8 Reproduction Operator . . . . . . . . . . . . . 458 11.8 Evolutionary Programming . . . . . . . . . . . . . . . 459 11.8.1 History . . . . . . . . . . . . . . . . . . . . . . 459 11.8.2 Procedure of Evolutionary Programming . . . 460 11.8.3 EPs and GAs. . . . . . . . . . . . . . . . . . . 461 11.8.4 Algorithm of EP . . . . . . . . . . . . . . . . . 461 11.8.5 Flowchart . . . . . . . . . . . . . . . . . . . . . 461 11.9 Evolutionary Strategies . . . . . . . . . . . . . . . . . 478 11.9.1 Solution Representation . . . . . . . . . . . . . 480 11.9.2 Mutation . . . . . . . . . . . . . . . . . . . . . 480 11.9.3 Recombination . . . . . . . . . . . . . . . . . . 481 11.9.4 Population Assessment . . . . . . . . . . . . . 482 11.9.5 Convergence Criteria . . . . . . . . . . . . . . 483 11.9.6 Computational Considerations . . . . . . . . . 483 11.9.7 Algorithm Performance . . . . . . . . . . . . . 484 11.10 Advantages and Disadvantagesof EvolutionaryCompu- tation . . . . . . . . . . . . . . . . . . . . . . . . . . . 485 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . 488 12 EvolutionaryAlgorithmsImplementedUsingMATLAB 491 12.1 Illustration 1: Differential Evolution Optimizer . . . . 491 12.2 Illustration 2: Design of a Proportional-Derivative Con- troller Using Evolutionary Algorithm for Tanker Ship Heading Regulation . . . . . . . . . . . . . . . . . . . 502 © 2010 by Taylor and Francis Group, LLC