Springer Series in Bio-/Neuroinformatics 4 Petia Koprinkova-Hristova Valeri Mladenov Nikola K. Kasabov Editors Artificial Neural Networks Methods and Applications in Bio-/Neuroinformatics Springer Series in Bio-/Neuroinformatics Volume 4 Serieseditor NikolaK.Kasabov,AucklandUniversityofTechnology,Auckland,NewZealand e-mail:[email protected] Aims&Scope SpringerSeriesinBio-/Neuroinformaticspublishescontentonfundamentalprinci- plesaswellasstate-of-arttheories,methodsandapplicationsintherapidlyevolv- ing fields of bioinformatics and neuroinformatics. The series is unique in that it integrates three interdependent disciplines, namely information science, bioinfor- matics and neuroinformatics. The series covers general informatics methods and techniques,likedatamining,databasemanagementsystems,machinelearning,ar- tificial neuralnetworks, evolutionarycomputation,chaostheory,quantumcompu- tation,statistical methods,andimageandsignalprocessing,aswellastheirappli- cationstodifferentareasofresearch,includingbigdata,functionalanalysisofthe brain,computationalsystemsbiology,computationalsystemsneuroscience,health informatics, personalizedmedicine, rehabilitationmedicine, industrial biotechnol- ogy, and many more. The series publishes monographs, contributed volumes and conferenceproceedings,aswellasadvancedtextbooks. AdvisoryBoard Shun-ichiAmari,RIKENBrainScienceInstitute,Japan PaoloAvesani,FondazioneBrunoKessler,Trento,Italy LubicaBenuskova,UniversityofOtago,NewZealand ChrisM.Brown,UniversityofOtago,NewZealand RichardJ.Duro,UniversityofCoruña,Spain PetiaGeorgieva,UniversityofAveiro,Portugal KaizhuHaung,ChineseAcademyofSciences,Beijing,China Zeng-GuangHou,ChineseAcademyofSciences,Beijing,China GiacomoIndiveri,ETHandUniversityofZurich,Switzerland IrwinKing,TheChineseUniversityofHongKong HiroshiKojima,TamagawaUniversity,Japan RobertKozma,TheUniversityofMemphis,USA AndreasKönig,TechnicalUniversityofKeiserlautern,Germany DaniloMandic,ImperialCollegeLondon,UnitedKingdom FrancescoMasulli,UniversityofGenova,Italy MartinMcGinnity,UniversityofUlster,N.Ireland HeikeSichtig,UniversityofFlorida,Gainesville,Florida,USA Jean-PhilippeThivierge,UniversityofOttawa,Canada ShiroUsui,RIKENBrainScienceInstitute,Japan AlessandroE.P.Villa,UniversityofLausanne,Switzerland JieYang,ShanghaiJiaotongUniversity,China Moreinformationaboutthisseriesathttp://www.springer.com/series/10088 · Petia Koprinkova-Hristova Valeri Mladenov Nikola K. Kasabov Editors Artificial Neural Networks Methods and Applications in Bio-/Neuroinformatics ABC Editors PetiaKoprinkova-Hristova NikolaK.Kasabov InstituteofInformationandCommunication AucklandUniversityofTechnology Technologies Auckland BulgarianAcademyofSciences NewZealand Sofia Bulgaria ValeriMladenov TechnicalUniversitySofia Sofia Bulgaria ISSN2193-9349 ISSN2193-9357 (electronic) ISBN978-3-319-09902-6 ISBN978-3-319-09903-3 (eBook) DOI10.1007/978-3-319-09903-3 LibraryofCongressControlNumber:2014945765 SpringerChamHeidelbergNewYorkDordrechtLondon (cid:2)c SpringerInternationalPublishingSwitzerland2015 Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation, broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology nowknownorhereafterdeveloped.Exemptedfromthislegalreservationarebriefexcerptsinconnection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher’slocation,initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer. PermissionsforusemaybeobtainedthroughRightsLinkattheCopyrightClearanceCenter.Violations areliabletoprosecutionundertherespectiveCopyrightLaw. Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant protectivelawsandregulationsandthereforefreeforgeneraluse. Whiletheadviceandinformationinthisbookarebelievedtobetrueandaccurateatthedateofpub- lication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityforany errorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,withrespect tothematerialcontainedherein. Printedonacid-freepaper SpringerispartofSpringerScience+BusinessMedia(www.springer.com) Preface The edited bookincludeschaptersthatpresentselected and extendedpapersfrom theInternationalConferenceonArtificialNeuralNetworks(ICANN)2013.Founded in1991,theICANNbecomethePremierannualconferenceoftheEuropeanNeural NetworkSociety.Itsmaingoalistobringtogetherandtofacilitatecontactsbetween researchersfrominformationsciencesandneurosciencesandtoprovideahigh-level internationalforumforbothacademicandindustrialcommunities. The selected and invited to the present book chapters were selected among ICANN papers that received highest scores during the strong peer review assess- ment(almost40%rejectionrate)andamongselectedbysessionchairsbestpresen- tations. The collected in the book chapters are presented in topical sections that cover wide range of contemporary topics varying from neural network theory and models, machine learning and learning algorithms, brain-machine interaction and bio-inspiredsystems,patternrecognitionandclassificationaswellasvariousappli- cations. Thebookchaptersincludenewtheoreticaldevelopmentsinrecurrentneuralnet- works and reservoir computing, new and improved training algorithms for Deep Boltzmann Machines (DBM), tapped delay feedforward architectures and kernel machines, reinforcement learning and Adaptive Critic Designs (ACD), new bio- inspired models and architectures related to cell assembly mechanisms, visual perception and natural language understanding,new and improvedalgorithms for pattern recognition with applications to gesture classification, handwritten digit recognitionandtimeseriesforecasting. The book will be of interest to all researchersand postgraduatestudents in the area of computational intelligence, applied mathematics, computer science, engi- neering,neuroscience,andotherrelatedareas. June2014 Editors SofiaandAuckland PetiaKoprinkova-Hristova ValeriMladenov NikolaK.Kasabov Contents Neural Networks Theory and Models Recurrent Neural Networks and Super-Turing Interactive Computation .................................................. 1 JérémieCabessa,AlessandroE.P.Villa ImageClassificationwithNonnegativeMatrixFactorizationBasedon SpectralProjectedGradient ..................................... 31 RafałZdunek,AnhHuyPhan,AndrzejCichocki Energy-TimeTradeoffinRecurrentNeuralNets .................... 51 JirˇíŠíma AnIntroductiontoDelay-CoupledReservoirComputing ............. 63 JohannesSchumacher,HazemToutounji,GordonPipa Double-LayerVectorPerceptronforBinaryPatternsRecognition...... 91 VladimirKryzhanovskiy,IrinaZhelavskaya Local Detectionof Communities by AttractorNeural-Network Dynamics ..................................................... 115 HiroshiOkamoto LearningGestaltFormationsforOscillatorNetworks ................ 127 MartinMeier,RobertHaschke,HelgeJ.Ritter Analysingthe Multiple TimescaleRecurrentNeuralNetworkfor EmbodiedLanguageUnderstanding .............................. 149 StefanHeinrich,SvenMagg,StefanWermter LearningtoLookandLookingtoRemember:ANeural-Dynamic EmbodiedModelforGenerationofSaccadicGazeShiftsandMemory Formation .................................................... 175 YuliaSandamirskaya,TobiasStorck VIII Contents New Machine Learning Algorithms forNeural Networks HowtoPretrainDeepBoltzmannMachinesinTwoStages ............ 201 KyunghyunCho,TapaniRaiko,AlexanderIlin,JuhaKarhunen TrainingDynamicNeuralNetworksUsingthe ExtendedKalman FilterforMulti-Step-AheadPredictions............................ 221 ArtemChernodub LearningasConstraintReactions................................. 245 GiorgioGnecco,MarcoGori,StefanoMelacci,MarcelloSanguineti Baseline-FreeSamplinginParameterExploringPolicyGradients: SuperSymmetricPGPE......................................... 271 FrankSehnke,TingtingZhao Sparse Approximations to Value Functions in Reinforcement Learning...................................................... 295 HunorS.Jakab,LehelCsató NeuralNetworksSolutionofOptimalControlProblemswithDiscrete TimeDelaysandTime-DependentLearningofInfinitesimalDynamic System ....................................................... 315 TiborKmet,MariaKmetova PatternRecognition, Classificationand OtherNeural NetworkApplications Applying PrototypeSelection and Abstraction Algorithms for EfficientTime-SeriesClassification................................ 333 StefanosOugiaroglou,LeonidasKaramitopoulos,Christos Tatoglou, GeorgiosEvangelidis,DimitrisA.Dervos EnforcingGroupStructurethroughtheGroupFusedLasso........... 349 CarlosM.Alaíz,ÁlvaroBarbero,JoséR.Dorronsoro IncrementalAnomaly Identificationin Flight Data Analysis by AdaptedOne-ClassSVMMethod................................. 373 DenisKolev,MikhailSuvorov,EvgeniyMorozov,GareginMarkarian, PlamenAngelov InertialGestureRecognitionwithBLSTM-RNN .................... 393 Grégoire Lefebvre, Samuel Berlemont, Franck Mamalet, ChristopheGarcia OnlineRecognitionofFixations,Saccades,andSmoothPursuitsfor AutomatedAnalysisofTrafficHazardPerception ................... 411 Enkelejda Kasneci, Gjergji Kasneci, Thomas C. Kübler, WolfgangRosenstiel Contents IX Input Transformationand Output Combination for Improved HandwrittenDigitRecognition ................................... 435 Juan M. Alonso-Weber, M. Paz Sesmero, German Gutierrez, AgapitoLedezma,AraceliSanchis FeatureSelectionforIntervalForecastingofElectricityDemandTime SeriesData.................................................... 445 MashudRana,IrenaKoprinska,AbbasKhosravi Stacked Denoising Auto-Encoders for Short-Term Time Series Forecasting.................................................... 463 PabloRomeu,FranciscoZamora-Martínez,PalomaBotella-Rocamora, JuanPardo AuthorIndex ..................................................... 487 Recurrent Neural Networks and Super-Turing Interactive Computation Je´re´mieCabessaandAlessandroE.P.Villa Abstract.Wepresentacompleteoverviewofthecomputationalpowerofrecurrent neural networks involved in an interactive bio-inspired computational paradigm. More precisely, we recall the results stating that interactive rational- and real- weightedneuralnetworksareTuring-equivalentandsuper-Turing,respectively.We furtherprovethatinteractiveevolvingneuralnetworksaresuper-Turing,irrespective of whethertheir synaptic weights are modeledby rationalor real numbers. These resultsshowthatthecomputationalpowersofneuralnetsinvolvedinaclassicalor in an interactivecomputationalframeworkfollowsimilar patternsof characteriza- tion.Theysuggestthatsomeintrinsiccomputationalcapabilitiesofthebrainmight liebeyondthescopeofTuring-equivalentmodelsofcomputation,hencesurpassthe potentialitieseverycurrentstandardartificialmodelsofcomputation. 1 Introduction Understanding the computational and dynamical capabilities of biological neural networks represents an issue of central importance. In this context, much interest has been focused on comparing the computational powers of diverse theoretical neuralmodelswith those of abstract computingdevices.Nowadays, the computa- tional capabilitiesof neuralmodels is knownto be tightly related to the nature of the activation functionof the neurons,to the nature of their synaptic connections, totheeventualpresenceofnoiseinthemodel,tothepossibilityforthenetworksto evolveovertime,andtothecomputationalparadigmperformedbythenetworks. Je´re´mieCabessa LaboratoryofMathematicalEconomics(LEMMA),UniversityofParis2–Panthe´on-Assas, 4RueBlaiseDesgoffe,75006Paris,France e-mail:[email protected] AlessandroE.P.Villa NeuroheuristicResearchGroup,FacultyofBusinessandEconomics,UniversityofLausanne, CH-1015Lausanne,Switzerland e-mail:[email protected] (cid:2)c SpringerInternationalPublishingSwitzerland2015 1 P.Koprinkova-Hristovaetal.(eds.),ArtificialNeuralNetworks, SpringerSeriesinBio-/Neuroinformatics4,DOI:10.1007/978-3-319-09903-3_1