T-Labs Series in Telecommunication Services Gabriel Mittag Deep Learning Based Speech Quality Prediction T-Labs Series in Telecommunication Services SeriesEditors SebastianMo¨ller,QualityandUsabilityLab,TechnischeUniversitätBerlin,Berlin, Germany Axel Ku¨pper, Telekom Innovation Laboratories, Technische Universität Berlin, Berlin,Germany AlexanderRaake,AudiovisualTechnologyGroup,TechnischeUniversitätIlmenau, Ilmenau,Germany It is the aim of the Springer Series in Telecommunication Services to foster an interdisciplinary exchange of knowledge addressing all topics which are essential for developing high-quality and highly usable telecommunication services. This includes basic concepts of underlying technologies, distribution networks, archi- tectures and platforms for service design, deployment and adaptation, as well as the users’perception of telecommunication services.By taking avertical perspec- tiveoverallthesesteps,weaimtoprovidethescientificbasesforthedevelopment and continuous evaluation of innovative services which provide a better value for their users. In fact, the human-centric design of high-quality telecommunication services – the so called “quality engineering” – forms an essential topic of this series,asitwillultimatelyleadtobetteruserexperienceandacceptance.Theseries isdirectedtowardsbothscientistsandpractitionersfromallrelateddisciplinesand industries. **Indexing:booksinthisseriesareindexinginScopus** Moreinformationaboutthisseriesathttps://link.springer.com/bookseries/10013 Gabriel Mittag Deep Learning Based Speech Quality Prediction GabrielMittag TechnischeUniversita¨tBerlin Berlin,Germany ISSN2192-2810 ISSN2192-2829 (electronic) T-LabsSeriesinTelecommunicationServices ISBN978-3-030-91478-3 ISBN978-3-030-91479-0 (eBook) https://doi.org/10.1007/978-3-030-91479-0 ©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNatureSwitzerland AG2022 Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuse ofillustrations,recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,and transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar ordissimilarmethodologynowknownorhereafterdeveloped. Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant protectivelawsandregulationsandthereforefreeforgeneraluse. Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthisbook arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional claimsinpublishedmapsandinstitutionalaffiliations. ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland Preface Instrumentalspeechqualitypredictionisalong-studiedfieldinwhichmanymodels have been presented. However, in particular, the single-ended prediction without the use of a clean reference signal remains challenging. This book studies how recent developments in machine learning can be leveraged to improve the quality prediction of transmitted speech and additionally provide diagnostic information through the prediction of speech quality dimensions. In particular, different deep learning architectures were analyzed towards their suitability to predict speech quality. To this end, a large dataset with distorted speech files and crowdsourced subjective ratings was created. A number of deep learning architectures, such as CNNs, LSTM networks, and Transformer/self-attention networks, were combined andcompared.ItwasfoundthatanetworkwithCNN,self-attention,andaproposed attention-pooling delivers the best single-ended speech quality predictions on the considered dataset. Furthermore, a double-ended speech quality prediction model basedonaSiameseneuralnetworkispresented.However,itcouldbeshownthat, in contrast to traditional models, deep learning models only slightly benefit from includingthecleanreferencesignal.Forthepredictionofperceptualspeechquality dimensions,amulti-tasklearningbasedmodelispresentedthatpredictstheoverall speech quality and the quality dimensions noisiness, coloration, discontinuity, and loudnessinparallel,wheremostoftheneuralnetworklayersaresharedbetweenthe individualtasks.Finally,thesingle-endedspeechqualitypredictionmodelNISQA ispresentedthatwastrainedonalargevarietyof59differentdatasets.Becausethe trainingdatasetscomefromavarietyofsourcesandcontaindifferentqualityranges, they are exposed to subjective biases. Therefore, the same speech distortions can leadtoverydifferentqualityratingsintwodatasets.Topreventanegativeinfluence of this effect, a bias-aware loss function is proposed that estimates and considers the biases during the training of the neural network weights. The final model was testedonalive-talkingtestsetwithrealrecordedphonecalls,onwhichitachieved aPearson’scorrelationof0.90fortheoverallspeechqualityprediction. Berlin,Germany GabrielMittag v Acknowledgments I am very grateful to the many supporters who have made this work possible. Duringthelastyears,Ihadthepleasuretomeetandgettoknowmanyinteresting peopleattheQualityandUsabilityLab,butalsoatseveralacademicconferences, workshops, and ITU meetings. First, I would like to thank my thesis supervisor Prof. Dr. Sebastian Möller for his support, his scientific expertise, and his advice thatgreatlyhelpedmetowriteandcompletethisbook.Myspecialthanksalsogo toDr.FriedemannKöster,whointroducedmetotheexcitingfieldofspeechquality estimationandwithoutwhomIprobablywouldnothavestartedmydoctoralstudies. IwouldliketothankmystudentassistantLouisLiedtkeforhissupportandalsoall thestudentsIhadthepleasuretosuperviseduringtheirbachelor’sormaster’stheses preparation,inparticularAssmaaChehadiforherworkononeofthedatasetsused in this book and Huahua Maier on his work on the Android recording app. I also wanttothankProf.Dr.GerhardSchmidtandTobiasHübschenfromtheUniversity of Kiel and Dr. Jens Berger for the great collaboration during the DFG project. I would like to thank Prof. Tiago H. Falk. and, a second time, Prof. Dr. Gerhard Schmidtforreviewingthisbookandforservingonmydoctoralcommittee.Many thanks go to Irene Hube-Achter, Yasmin Hillebrenner, and Tobias Jettkowski for their excellent administrative and technical support. Thanks to all my former and current colleagues at the Quality and Usability Lab for the numerous discussions, exchangeofresearchideas,andforkeepingmecompanyduringmycoffeebreaks and making sure that it never got too boring at the lab, including Steven Schmidt, SamanZadtootaghaj,SaiSirishaRallabandi,ThiloMichael,TanjaKojic,Dr.Babak Naderi,Dr.LauraFernándezGallardo,Dr.PatrickEhrenbrink,Dr.DennisGuse,Dr. MaijaPoikela,Dr.FalkSchiffner,andDr.StefanUhrig,andmanymore.Thankyou allforagreattime! vii Contents 1 Introduction .................................................................. 1 1.1 Motivation............................................................... 1 1.2 ThesisObjectivesandResearchQuestions............................ 4 1.3 Outline................................................................... 4 2 QualityAssessmentofTransmittedSpeech............................... 7 2.1 SpeechCommunicationNetworks..................................... 7 2.2 SpeechQualityandSpeechQualityDimensions...................... 10 2.3 SubjectiveAssessment.................................................. 12 2.4 SubjectiveAssessmentviaCrowdsourcing............................ 16 2.5 TraditionalInstrumentalMethods...................................... 18 2.5.1 ParametricModels.............................................. 18 2.5.2 Double-EndedSignal-BasedModels .......................... 19 2.5.3 Single-EndedSignal-BasedModels ........................... 21 2.6 MachineLearningBasedInstrumentalMethods ...................... 22 2.6.1 Non-DeepLearningMachineLearningApproaches.......... 23 2.6.2 DeepLearningArchitectures................................... 24 2.6.3 DeepLearningBasedSpeechQualityModels................ 28 2.7 Summary ................................................................ 31 3 NeuralNetworkArchitecturesforSpeechQualityPrediction.......... 33 3.1 Dataset................................................................... 33 3.1.1 SourceFiles..................................................... 34 3.1.2 SimulatedDistortions........................................... 35 3.1.3 LiveDistortions................................................. 39 3.1.4 ListeningExperiment........................................... 40 3.2 OverviewofNeuralNetworkModel................................... 42 3.3 Mel-SpecSegmentation ................................................ 43 3.4 FramewiseModel....................................................... 43 3.4.1 CNN............................................................. 43 3.4.2 FeedforwardNetwork .......................................... 45 ix x Contents 3.5 Time-DependencyModelling .......................................... 46 3.5.1 LSTM ........................................................... 47 3.5.2 Transformer/Self-Attention .................................... 48 3.6 TimePooling............................................................ 51 3.6.1 Average-/Max-Pooling ......................................... 51 3.6.2 Last-Step-Pooling............................................... 51 3.6.3 Attention-Pooling............................................... 52 3.7 ExperimentsandResults................................................ 53 3.7.1 TrainingandEvaluationMetric................................ 53 3.7.2 FramewiseModel............................................... 54 3.7.3 Time-DependencyModel ...................................... 56 3.7.4 PoolingModel .................................................. 57 3.8 Summary ................................................................ 58 4 Double-EndedSpeechQualityPredictionUsingSiameseNetworks... 59 4.1 Introduction ............................................................. 59 4.2 Method................................................................... 60 4.2.1 SiameseNeuralNetwork....................................... 62 4.2.2 ReferenceAlignment........................................... 62 4.2.3 FeatureFusion .................................................. 64 4.3 Results................................................................... 65 4.3.1 LSTMvsSelf-Attention........................................ 65 4.3.2 Alignment....................................................... 66 4.3.3 FeatureFusion .................................................. 67 4.3.4 Double-EndedvsSingle-Ended................................ 68 4.4 Summary ................................................................ 70 5 PredictionofSpeechQualityDimensionswithMulti-Task Learning ...................................................................... 73 5.1 Introduction ............................................................. 73 5.2 Multi-TaskModels...................................................... 75 5.2.1 FullyConnected(MTL-FC).................................... 76 5.2.2 FullyConnected+Pooling(MTL-POOL) .................... 77 5.2.3 FullyConnected+Pooling+Time-Dependency (MTL-TD)....................................................... 78 5.2.4 FullyConnected+Pooling+Time-Dependency+ CNN(MTL-CNN).............................................. 79 5.3 Results................................................................... 79 5.3.1 Per-TaskEvaluation ............................................ 80 5.3.2 All-TasksEvaluation ........................................... 83 5.3.3 ComparingDimension ......................................... 84 5.3.4 DegradationDecomposition.................................... 85 5.4 Summary ................................................................ 87 Contents xi 6 Bias-AwareLossforTrainingfromMultipleDatasets................... 89 6.1 Method................................................................... 90 6.1.1 LearningwithBias-AwareLoss................................ 91 6.1.2 AnchoringPredictions.......................................... 92 6.2 ExperimentsandResults................................................ 93 6.2.1 SyntheticData .................................................. 94 6.2.2 MinimumAccuracyr ......................................... 95 th 6.2.3 TrainingExampleswithandWithoutAnchoring ............. 96 6.2.4 ConfigurationComparisons.................................... 97 6.2.5 SpeechQualityDataset......................................... 99 6.3 Summary ................................................................ 100 7 NISQA:ASingle-EndedSpeechQualityModel.......................... 103 7.1 Datasets.................................................................. 103 7.1.1 POLQAPool.................................................... 104 7.1.2 ITU-TPSuppl.23.............................................. 104 7.1.3 OtherDatasets .................................................. 106 7.1.4 Live-TalkingTestSet........................................... 109 7.2 ModelandTraining..................................................... 110 7.2.1 Model............................................................ 110 7.2.2 Bias-AwareLoss................................................ 111 7.2.3 HandlingMissingDimensionRatings......................... 111 7.2.4 Training ......................................................... 112 7.3 Results................................................................... 114 7.3.1 EvaluationMetrics.............................................. 115 7.3.2 ValidationSetResults:OverallQuality ....................... 116 7.3.3 ValidationSetResults:QualityDimensions................... 118 7.3.4 TestSetResults................................................. 122 7.3.5 ImpairmentLevelvsQualityPrediction....................... 125 7.4 Summary ................................................................ 138 8 Conclusions................................................................... 141 A DatasetConditionTables.................................................... 145 B TrainandValidationDatasetDimensionHistograms ................... 149 References......................................................................... 153 Index............................................................................... 161