Renewable Energy Series 17 Nonlinear Optimization in EN no gn i nl Electrical Engineering with i en ee ra Applications in MATLAB® inr gO p w t itim h Aiz a p t pio lin Nonlinear Optimization in Electrical Engineering with Applications in MATLAB® Mohamed Bakr is an Associate c provides an introductory course on nonlinear optimization in electrical engineering, Professor at the Department of a i with a focus on applications such as the design of electric, microwave, and Electrical and Computer Engineering, tn photonic circuits, wireless communications, and digital filter design. McMaster University, Canada, io where his research interests include E Basic concepts are introduced using a step-by-step approach and illustrated with optimization methods, computer- nle MATLAB® codes that the reader can use and adapt. Topics covered include: aided design and modelling of s c (cid:127) classical optimization methods mapipcrloicwataiovnes c, isrcmuiatsrt, anneaulryasli sn eotfw orks intr (cid:127) one dimensional optimization microwave circuits and efficient ic (cid:127) unconstrained and constrained optimization optimization using time domain Ma (cid:127)(cid:127) gsploabcael mopatpimpiinzagt ioopntimization simulation methods. Al (cid:127) adjoint variable methods T Nonlinear Optimization in L Nonlinear Optimization in Electrical Engineering with Applications in MATLAB® is A essential reading for advanced students in electrical engineering. B ® Electrical Engineering with Applications in MATLAB® B a k r Mohamed Bakr The Institution of Engineering and Technology www.theiet.org ISBN 978-1-84919-543-0 Nonlinear Optimization.indd 1 23/08/2013 09:19:27 Nonlinear Optimization in Electrical Engineering with ‡ Applications in MATLAB Nonlinear Optimization in Electrical Engineering with ‡ Applications in MATLAB Mohamed Bakr The Institution of Engineering andTechnology PublishedbyTheInstitutionofEngineeringandTechnology,London,UnitedKingdom TheInstitutionofEngineeringandTechnologyisregisteredasaCharityinEngland& Wales(no.211014)andScotland(no.SC038698). †TheInstitutionofEngineeringandTechnology2013 Firstpublished2013 ThispublicationiscopyrightundertheBerneConventionandtheUniversalCopyright Convention.Allrightsreserved.Apartfromanyfairdealingforthepurposesofresearch orprivatestudy,orcriticismorreview,aspermittedundertheCopyright,Designsand PatentsAct1988,thispublicationmaybereproduced,storedortransmitted,inany formorbyanymeans,onlywiththepriorpermissioninwritingofthepublishers,orin thecaseofreprographicreproductioninaccordancewiththetermsoflicencesissued bytheCopyrightLicensingAgency.Enquiriesconcerningreproductionoutsidethose termsshouldbesenttothepublisherattheundermentionedaddress: TheInstitutionofEngineeringandTechnology MichaelFaradayHouse SixHillsWay,Stevenage Herts,SG12AY,UnitedKingdom www.theiet.org Whiletheauthorandpublisherbelievethattheinformationandguidancegiveninthis workarecorrect,allpartiesmustrelyupontheirownskillandjudgementwhenmaking useofthem.Neithertheauthornorpublisherassumesanyliabilitytoanyoneforany lossordamagecausedbyanyerrororomissioninthework,whethersuchanerroror omissionistheresultofnegligenceoranyothercause.Anyandallsuchliabilityis disclaimed. Themoralrightsoftheauthortobeidentifiedasauthorofthisworkhavebeen assertedbyhiminaccordancewiththeCopyright,DesignsandPatentsAct1988. BritishLibraryCataloguinginPublicationData AcataloguerecordforthisproductisavailablefromtheBritishLibrary ISBN978-1-84919-543-0(hardback) ISBN978-1-84919-544-7(PDF) TypesetinIndiabyMPSLimited PrintedintheUKbyCPIGroup(UK)Ltd,Croydon To my wife Mahetab, my children Jannah, Omar, and Youssef, and to my parents to whom I am indebted for as long as I live Contents Preface xi Acknowledgments xv 1 Mathematical background 1 1.1 Introduction 1 1.2 Vectors 1 1.3 Matrices 3 1.4 The solution of linear systems of equations 6 1.5 Derivatives 11 1.5.1 Derivative approximation 11 1.5.2 The gradient 12 1.5.3 The Jacobian 14 1.5.4 Second-order derivatives 15 1.5.5 Derivatives of vectors and matrices 16 1.6 Subspaces 18 1.7 Convergence rates 20 1.8 Functionsand sets 20 1.9 Solutions of systems of nonlinear equations 22 1.10 Optimization problem definition 25 References 25 Problems 25 2 Anintroduction to linear programming 29 2.1 Introduction 29 2.2 Examples of linear programs 29 2.2.1 Afarming example 29 2.2.2 Aproductionexample 30 2.2.3 Power generation example 31 2.2.4 Wireless communication example 32 2.2.5 Abattery charging example 32 2.3 Standard form of an LP 33 2.4 Optimality conditions 37 2.5 The matrix form 39 2.6 Canonical augmented form 40 2.7 Moving from one basic feasible solution to another 42 2.8 Cost reduction 45 2.9 The classical Simplex method 46 2.10 Starting the Simplex method 49 2.10.1 Endless pivoting 51 2.10.2 The big M approach 51 2.10.3 The two-phase Simplex 52 viii NonlinearoptimizationinelectricalengineeringwithapplicationsinMATLAB(cid:2) 2.11 Advanced topics 55 A2.1 Minimax optimization 55 A2.1.1Minimax problem definition 55 A2.1.2Minimax solution usinglinear programming 57 A2.1.3Amicrowave filter example 59 A2.1.4The design of coupled microcavities optical filter 61 References 65 Problems 65 3 Classical optimization 69 3.1 Introduction 69 3.2 Single-variable Taylor expansion 69 3.3 Multidimensional Taylor expansion 71 3.4 Meaningof the gradient 73 3.5 Optimality conditions 76 3.6 Unconstrained optimization 76 3.7 Optimization with equality constraints 78 3.7.1 Method of direct substitution 79 3.7.2 Method of constrained variation 80 3.8 Lagrange multipliers 84 3.9 Optimization with inequality constraints 86 3.10 Optimization with mixed constraints 92 A3.1 Quadratic programming 92 A3.2 Sequential quadratic programming 95 References 99 Problems 99 4 One-dimensionaloptimization-Line search 101 4.1 Introduction 101 4.2 Bracketing approaches 102 4.2.1 Fixed line search 103 4.2.2 Accelerated line search 104 4.3 Derivative-free line search 105 4.3.1 Dichotomousline search 105 4.3.2 The interval-halving method 106 4.3.3 The Fibonacci search 108 4.3.4 The Golden Sectionmethod 111 4.4 Interpolationapproaches 112 4.4.1 Quadratic models 113 4.4.2 Cubic interpolation 116 4.5 Derivative-based approaches 119 4.5.1 The classical Newton method 119 4.5.2 Aquasi-Newtonmethod 121 4.5.3 The Secant method 122 4.6 Inexact line search 123 A4.1 Tuning of electric circuits 124 Contents ix A4.1.1Tuning of a current source 125 A4.1.2Coupling of nanowires 127 A4.1.3Matching of microwave antennas 128 References 129 Problems 130 5 Derivative-free unconstrainedtechniques 131 5.1 Why unconstrained optimization? 131 5.2 Classification of unconstrained optimization techniques 131 5.3 The random jump technique 132 5.4 The random walk method 133 5.5 Grid search method 134 5.6 The univariate method 135 5.7 The pattern search method 137 5.8 The Simplex method 140 5.9 Responsesurface approximation 143 A5.1 Electrical application: impedance transformers 146 A5.2 Electrical application: the designof photonic devices 149 References 151 Problems 152 6 First-order unconstrainedoptimization techniques 153 6.1 Introduction 153 6.2 The steepest descent method 153 6.3 The conjugate directions method 156 6.3.1 Definition of conjugacy 157 6.3.2 Powell’smethod of conjugate directions 158 6.4 Conjugate gradient methods 162 A6.1 Solution of large systems of linear equations 164 A6.2 The designof digital FIRfilters 169 References 173 Problems 173 7 Second-order unconstrainedoptimization techniques 175 7.1 Introduction 175 7.2 Newton’s method 175 7.3 The Levenberg–Marquardt method 178 7.4 Quasi-Newton methods 179 7.4.1 Broyden’s rank-1 update 180 7.4.2 The Davidon–Fletcher–Powell (DFP)formula 182 7.4.3 The Broyden–Fletcher–Goldfarb–Shanno method 184 7.4.4 The Gauss–Newton method 185 A7.1 Wireless channel characterization 188 A7.2 The parameter extractionproblem 189 A7.3 Artificial neural networks training 193 References 201 Problems 201