Uday Kamath · John Liu · James Whitaker Deep Learning for NLP and Speech Recognition Deep Learning for NLP and Speech Recognition Uday Kamath • John Liu • James Whitaker Deep Learning for NLP and Speech Recognition 123 UdayKamath JohnLiu DigitalReasoningSystemsInc. IntelluronCorporation McLean Nashville VA,USA TN,USA JamesWhitaker DigitalReasoningSystemsInc. McLean VA,USA ISBN978-3-030-14595-8 ISBN978-3-030-14596-5 (eBook) https://doi.org/10.1007/978-3-030-14596-5 ©SpringerNatureSwitzerlandAG2019 Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation, broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology nowknownorhereafterdeveloped. Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant protectivelawsandregulationsandthereforefreeforgeneraluse. Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor theeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinorforany errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional claimsinpublishedmapsandinstitutionalaffiliations. ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG. Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland TomyparentsKrishnaandBharathi,mywife Pratibha,thekidsAarohandBrandy,my familyandfriendsfortheirsupport. –UdayKamath ToCatherine,GabrielleKaili-May,Eugene andTinaforinspiringmealways. –JohnLiu TomymotherNancyforherconstant support,myfamily,andmyfriendswhohave blessedmylifewithlove. –JamesWhitaker Foreword The publication of this book is a perfect timing. Existing books on deep learning either focus on theoretical aspects or are largely manuals for tools. But this book presentsanunprecedentedanalysisandcomparisonofdeeplearningtechniquesfor natural language and speech processing, closing the substantial gap between the- oryandpractice.Eachchapterdiscussesthetheoryunderpinningthetopics,andan exceptionalcollectionof13casestudiesindifferentapplicationareasispresented. They include classification via distributed representation, summarization, machine translation,sentimentanalysis,transferlearning,multitaskNLP,end-to-endspeech, andquestionanswering.Eachcasestudyincludestheimplementationandcompar- ison of state-of-the-art techniques, and the accompanying website provides source codeanddata.Thisisextraordinarilyvaluableforpractitioners,whocanexperiment firsthand with the methods and can deepen their understanding of the methods by applyingthemtoreal-worldscenarios. Thisbookoffersacomprehensivecoverageofdeeplearning,fromitsfoundations toadvancedandrecenttopics,includingwordembedding,convolutionalneuralnet- works, recurrent neural networks, attention mechanisms, memory-augmented net- works,multitasklearning,domainadaptation,andreinforcementlearning.Thebook isagreatresourceforpractitionersandresearchersbothinindustryandacademia, andthediscussedcasestudiesandassociatedmaterialcanserveasinspirationfora varietyofprojectsandhands-onassignmentsinaclassroomsetting. AssociateProfessoratGMU CarlottaDomeniconi,PhD Fairfax,VA,USA February2019 Naturallanguageandspeechprocessingapplicationssuchasvirtualassistantsand smart speakers play an important and ever-growing role in our lives. At the same time, amid an increasing number of publications, it is becoming harder to iden- tifythemostpromisingapproaches.AstheChiefAnalyticsOfficeratDigitalRea- soning and with a PhD in Big Data Machine Learning, Uday has access to both the practical and research aspects of this rapidly growing field. Having authored vii viii Foreword MasteringJavaMachineLearning,heisuniquelysuitedtobreakdownbothpracti- calandcutting-edgeapproaches.Thisbookcombinesboththeoreticalandpractical aspectsofmachinelearninginarareblend.Itconsistsofanintroductionthatmakes itaccessibletopeoplestartinginthefield,anoverviewofstate-of-the-artmethods that should be interesting even to people working in research, and a selection of hands-onexamplesthatgroundthematerialinreal-worldapplicationsanddemon- strateitsusefulnesstoindustrypractitioners. ResearchScientistatDeepMind SebastianRuder,PhD London,UK February2019 A few years ago, I picked up a few text-books to study topics related to arti- ficial intelligence—such as natural language processing and computer vision. My memory of reading these text-books largely consisted of staring helplessly out of thewindow.WheneverIattemptedtoimplementthedescribedconceptsandmath, I wouldn’t know where to start. This is fairly common in books written for aca- demicpurposes;theymockinglyleavetheactualimplementation“asanexerciseto the reader.” There are a few exceptional books that try to bridge this gap, written by people who know the importance of going beyond the math all the way to a working system. This book is one of those exceptions—with it’s discussions, case studies,codesnippets,andcomprehensivereferences,itdelightfullybridgesthegap betweenlearninganddoing. IespeciallyliketheuseofPythonandopen-sourcetoolsoutthere.It’sanopin- ionated take on implementing machine learning systems—one might ask the fol- lowingquestion:“WhynotX,”whereXcouldbeJava,C++,orMatlab?However, I find solace in the fact that it’s the most popular opinion, which gives the read- ers an immense support structure as they implement their own ideas. In the mod- ern Internet-connected world, joining a popular ecosystem is equivalent to having thousandsofhumansconnectingtogethertohelpeachother—fromStackOverflow posts solving an error message to GitHub repositories implementing high-quality systems.Togiveyouperspective,I’veseentheotherside,supportinganichecom- munityofenthusiastsinmachinelearningusingtheprogramminglanguageLuafor several years. It was a daily struggle to do new things—even basic things such as makingabarchart—preciselybecauseourcommunityofpeoplewasafeworders ofmagnitudesmallerthanPython’s. Overall,Ihopethereaderenjoysamodern,practicaltakeondeeplearningsys- tems, leveraging open-source machine learning systems heavily, and being taught a lot of “tricks of the trade” by the incredibly talented authors, one of whom I’ve knownforyearsandhaveseenbuildrobustspeechrecognitionsystems. ResearchEngineeratFacebookAIResearch(FAIR) SoumithChintala,PhD NewYork,NY,USA February2019 Preface WhyThisBook? Withthewidespreadadoptionofdeeplearning,naturallanguageprocessing(NLP), and speech applications in various domains such as finance, healthcare, and gov- ernmentandacrossourdailylives,thereisagrowingneedforonecomprehensive resource that maps deep learning techniques to NLP and speech and provides in- sights into using the tools and libraries for real-world applications. Many books focus on deep learning theory or deep learning for NLP-specific tasks, while oth- ersarecookbooksfortoolsandlibraries.But,theconstantfluxofnewalgorithms, tools,frameworks,andlibrariesinarapidlyevolvinglandscapemeansthatthereare few available texts that contain explanations of the recent deep learning methods andstate-of-the-artapproachesapplicabletoNLPandspeech,aswellasreal-world casestudieswithcodetoprovidehands-onexperience.Asanexample,youwould find it difficult to find a single source that explains the impact of neural attention techniques applied to a real-world NLP task such as machine translation across a range of approaches, from the basic to the state-of-the-art. Likewise, it would be difficulttofindasourcethatincludesaccompanyingcodebasedonwell-knownli- brarieswithcomparisonsandanalysisacrossthesetechniques. Thisbookprovidesthefollowingallinoneplace: • A comprehensive resource that builds up from elementary deep learning, text, andspeechprinciplestoadvancedstate-of-the-artneuralarchitectures • AreadyreferencefordeeplearningtechniquesapplicabletocommonNLPand speechrecognitionapplications • Ausefulresourceonsuccessfularchitecturesandalgorithmswithessentialmath- ematicalinsightsexplainedindetail • An in-depth reference and comparison of the latest end-to-end neural speech processingapproaches ix x Preface • Apanoramicresourceonleading-edgetransferlearning,domainadaptation,and deepreinforcementlearningarchitecturesfortextandspeech • Practicalaspectsofusingthesetechniqueswithtipsandtricksessentialforreal- worldapplications • Ahands-onapproachinusingPython-basedlibrariessuchasKeras,TensorFlow, andPyTorchtoapplythesetechniquesinthecontextofreal-worldcasestudies Inshort,theprimarypurposeofthisbookistoprovideasinglesourcethataddresses thegapbetweentheoryandpracticeusingcasestudieswithcode,experiments,and supportinganalysis. WhoIsThisBookfor? This book is intended to introduce the foundations of deep learning, natural lan- guageprocessing,andspeech,withanemphasisonapplicationandpracticalexpe- rience.ItisaimedatNLPpractitioners,graduatestudentsinEngineeringandCom- puter Science, advanced undergraduates, and anyone with the appropriate mathe- maticalbackgroundwhoisinterestedinanin-depthintroductiontotherecentdeep learningapproachesinNLPandspeech.Mathematically,weexpectthatthereader is familiar with multivariate calculus, probability, linear algebra, and Python pro- gramming. Pythonisbecomingthelinguafrancaofdatascientistsandresearchersforper- formingexperimentsindeeplearning.TherearemanylibrarieswithPython-enabled bindings for deep learning, NLP, and speech that have sprung up in the last few years. Therefore, we useboth the Python language and its accompanying libraries for all case studies in this book. As it is unfeasible to fully cover every topic in a single book, we present what we believe are the key concepts with regard to NLP andspeechthatwilltranslateintoapplication.Inparticular,wefocusontheinter- sectionofthoseareas,whereinwecanleveragedifferentframeworksandlibraries toexploremodernresearchandrelatedapplications. WhatDoesThisBookCover? The book is organized into three parts, aligning to different groups of readers and theirexpertise.Thethreepartsare: • Machine Learning, NLP, and Speech Introduction. The first part has three chapters that introduce readers to the fields of NLP, speech recognition, deep learning, and machine learning with basic hands-on case studies using Python- basedtoolsandlibraries. Preface xi • DeepLearningBasics.Thefivechaptersinthesecondpartintroducedeeplearn- ingandvarioustopicsthatarecrucialforspeechandtextprocessing,including wordembeddings,convolutionalneuralnetworks,recurrentneuralnetworks,and speechrecognitionbasics. • Advanced Deep Learning Techniques for Text and Speech. The third part hasfivechaptersthatdiscussthelatestresearchintheareasofdeeplearningthat intersectwithNLPandspeech.Topicsincludingattentionmechanisms,memory- augmented networks, transfer learning, multitask learning, domain adaptation, reinforcementlearning,andend-to-enddeeplearningforspeechrecognitionare coveredusingcasestudies. Next,wesummarizethetopicscoveredineachchapter. • IntheIntroduction,weintroducethereaderstothefieldsofdeeplearning,NLP, andspeechwithabriefhistory.Wepresentthedifferentareasofmachinelearn- inganddetaildifferentresourcesrangingfrombookstodatasetstoaidreadersin theirpracticaljourney. • The Basics of Machine Learning chapter provides a refresher of basic theory and important practical concepts. Topics covered include the learning process, supervisedlearning,datasampling,validationtechniques,overfittingandunder- fittingofthemodels,linearandnonlinearmachinelearningalgorithms,andse- quencedatamodeling.Thechapterendswithadetailedcasestudyusingstruc- tureddatatobuildpredictivemodelsandanalyzeresultsusingPythontoolsand libraries. • IntheTextandSpeechBasicschapter,weintroducethefundamentalsofcom- putationallinguisticsandNLPtothereader,includinglexical,syntactic,seman- tic,anddiscourserepresentations.Weintroducelanguagemodelinganddiscuss applicationssuchastextclassification,clustering,machinetranslation,question answering, automatic summarization, and automated speech recognition, con- cludingwithacasestudyontextclusteringandtopicmodeling. • TheBasicsofDeepLearningchapterbuildsuponthemachinelearningfounda- tionbyintroducingdeeplearning.Thechapterbeginswithafundamentalanal- ysis of the components of deep learning in the multilayer perceptron (MLP), followedbyvariationsonthebasicMLParchitectureandtechniquesfortraining deepneuralnetworks.Asthechapterprogresses,itintroducesvariousarchitec- tures for both supervised and unsupervised learning, such as multiclass MLPs, autoencoders, and generative adversarial networks (GANs). Finally, the mate- rialiscombinedintothecasestudy,analyzingbothsupervisedandunsupervised neuralnetworkarchitecturesonaspokendigitdataset. • For the Distributed Representations chapter, we investigate distributional semantics and word representations based on vector space models such as word2vec and GloVe. We detail the limitations of word embeddings including antonymy and polysemy and the approaches that can overcome them. We also