The Springer Series on Challenges in Machine Learning Sergio Escalera Ralf Herbrich Editors The NeurIPS ’18 Competition From Machine Learning to Intelligent Conversations The Springer Series on Challenges in Machine Learning Serieseditors HugoJairEscalante,AstrofisicaOpticayElectronica,INAOE,Puebla,Mexico IsabelleGuyon,ChaLearn,Berkeley,CA,USA SergioEscalera ,UniversitatdeBarcelonaandComputerVisionCenter, Barcelona,Spain Thebooksinthisinnovativeseriescollectpaperswritteninthecontextofsuccessful competitions in machine learning. They also include analyses of the challenges, tutorial material, dataset descriptions, and pointers to data and software. Together with the websites of the challenge competitions, they offer a complete teaching toolkitandavaluableresourceforengineersandscientists. Moreinformationaboutthisseriesathttp://www.springer.com/series/15602 Sergio Escalera • Ralf Herbrich Editors The NeurIPS ’18 Competition From Machine Learning to Intelligent Conversations 123 Editors SergioEscalera RalfHerbrich UniversitatdeBarcelonaandComputer Amazon(Berlin) VisionCenter Berlin,Berlin,Germany Barcelona,Spain ISSN2520-131X ISSN2520-1328 (electronic) TheSpringerSeriesonChallengesinMachineLearning ISBN978-3-030-29134-1 ISBN978-3-030-29135-8 (eBook) https://doi.org/10.1007/978-3-030-29135-8 ©SpringerNatureSwitzerlandAG2020 Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation, broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology nowknownorhereafterdeveloped. Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant protectivelawsandregulationsandthereforefreeforgeneraluse. Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional claimsinpublishedmapsandinstitutionalaffiliations. ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG. Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland Contents AGuidetotheNeurIPS2018Competitions .................................. 1 RalfHerbrichandSergioEscalera Pommerman&NeurIPS2018.................................................. 11 CinjonResnick,ChaoGao,GörögMárton,TakayukiOsogami,Liang Pang,andToshihiroTakahashi TheAIDrivingOlympicsatNeurIPS2018 ................................... 37 JulianZilly,JacopoTani,BreandanConsidine,BhairavMehta, AndreaF.Daniele,ManfredDiaz,GianmarcoBernasconi,ClaudioRuch, JanHakenberg,FlorianGolemo,A.KirstenBowser,MatthewR.Walter, RuslanHristov,SunilMallya,EmilioFrazzoli,AndreaCensi, andLiamPaull ArtificialIntelligenceforProsthetics:ChallengeSolutions.................. 69 ŁukaszKidzin´ski,CarmichaelOng,SharadaPrasannaMohanty, JenniferHicks,SeanCarroll,BoZhou,HongshengZeng,FanWang, RongzhongLian,HaoTian,WojciechJas´kowski,GarrettAndersen, OddRuneLykkebø,NihatEnginToklu,PranavShyam, RupeshKumarSrivastava,SergeyKolesnikov,OleksiiHrinchuk, AntonPechenko,MattiasLjungström,ZhenWang,XuHu, ZehongHu,MinghuiQiu,JunHuang,AlekseiShpilman,IvanSosin, OlegSvidchenko,AleksandraMalysheva,DanielKudenko,LanceRane, Aditya Bhatt, Zhengfei Wang, Penghui Qi, Zeyang Yu, Peng Peng, QuanYuan,WenxinLi,YunshengTian,RuihanYang,PingchuanMa, ShauhardaKhadka,SomdebMajumdar,ZachDwiel,YinyinLiu,Evren Tumer,JeremyWatson,MarcelSalathé,SergeyLevine,andScottDelp v vi Contents AdversarialVisionChallenge................................................... 129 WielandBrendel,JonasRauber,AlexeyKurakin,NicolasPapernot, Behar Veliqi, Sharada P. Mohanty, Florian Laurent, Marcel Salathé, MatthiasBethge,YaodongYu,HongyangZhang,SusuXu, HongbaoZhang,PengtaoXie,EricP.Xing,ThomasBrunner,Frederik Diehl,JérômeRony,LuizGustavoHafemann,ShuyuCheng,Yinpeng Dong,XuefeiNing,WenshuoLi,andYuWang TheInclusiveImagesCompetition............................................. 155 JamesAtwood,YoniHalpern,PallaviBaljekar,EricBreck, D.Sculley,PavelOstyakov,SergeyI.Nikolenko,IgorIvanov, RomanSolovyev,WeiminWang,andMihaSkalic TheSecondConversationalIntelligenceChallenge(ConvAI2)............. 187 EmilyDinan,VarvaraLogacheva,ValentinMalykh,AlexanderMiller, KurtShuster,JackUrbanek,DouweKiela,ArthurSzlam,IulianSerban, RyanLowe,ShrimaiPrabhumoye,AlanW.Black,AlexanderRudnicky, JasonWilliams,JoellePineau,MikhailBurtsev,andJasonWeston AutoML@NeurIPS2018Challenge:DesignandResults .................. 209 Hugo Jair Escalante, Wei-Wei Tu, Isabelle Guyon, Daniel L. Silver, EvelyneViegas,YuqiangChen,WenyuanDai,andQiangYang TheTrackingMachineLearningChallenge:AccuracyPhase.............. 231 Sabrina Amrouche, Laurent Basara, Paolo Calafiura, Victor Estrade, StevenFarrell,DiogoR.Ferreira,LiamFinnie,NicoleFinnie, CécileGermain,VladimirVavaGligorov,TobiasGolling, SergeyGorbunov,HeatherGray,IsabelleGuyon,MikhailHushchyn, VincenzoInnocente,MoritzKiehn,EdwardMoyse,Jean-FrançoisPuget, YuvalReina,DavidRousseau,AndreasSalzburger,AndreyUstyuzhanin, Jean-RochVlimant,JohanSokratesWind,TrianXylouris, andYetkinYilmaz Efficient and Robust Learning on Elaborated Gaits with CurriculumLearning............................................................ 265 BoZhou,HongshengZeng,FanWang,RongzhongLian,andHaoTian ConvAI2DatasetofNon-goal-OrientedHuman-to-BotDialogues......... 277 VarvaraLogacheva,ValentinMalykh,AlekseyLitinsky, andMikhailBurtsev Lost in Conversation: A Conversational Agent Based on the TransformerandTransferLearning........................................... 295 SergeyGolovanov,AlexanderTselousov,RaufKurbanov, andSergeyI.Nikolenko Contents vii AutomaticallyOptimizedGradientBoostingTreesforClassifying LargeVolumeHighCardinalityDataStreamsUnderConceptDrift...... 317 JobinWilson,AmitKumarMeher,BivinVinodkumarBindu, SantanuChaudhury,BrejeshLall,ManojSharma,andVishakhaPareek Index............................................................................... 337 A Guide to the NeurIPS 2018 Competitions RalfHerbrichandSergioEscalera Abstract Competitionshavebecomeanintegralpartofadvancingstate-of-the-art inartificialintelligence(AI).Theyexhibitoneimportantdifferencetobenchmarks: Competitionstestasystemend-to-endratherthanevaluatingonlyasinglecompo- nent;theyassessthepracticabilityofanalgorithmicsolutioninadditiontoassessing feasibility. In this volume, we present the details of eight competitions in the area of AI which took place between February to December 2018 and were presented at the Neural Information Processing Systems conference in Montreal, Canada on December8,2018.ThecompetitionsrangedfromchallengesinRobotics,Computer Vision,NaturalLanguageProcessing,Games,Health,SystemstoPhysics. 1 The ImportanceofCompetitions The field of artificial intelligence (AI) has seen a significant surge of interest and impactinsocietyoverthepast10years.Eventhoughsomeofthemostsuccessful methods in the field of AI have been studied and discovered several decades ago [10, 12], it was benchmarks and competitions who were the catalyst for proving thesemethodsinpractice.Startingfromtheby-nowclassicalMNIST1 andUSPS2 digitrecognitionbenchmarkdatasets—whichdroveresearchinclassificationlearn- 1TheMNISTdatabaseofhandwrittendigitshasatrainingsetof60,000examples,andatestsetof 10,000examplesof28×28grayscaleimagesdepictingletterszerotonine. 2Thedatasethas7291trainand2007testimagesof16×16grayscalepixelsdepictingletterszero tonine. R.Herbrich Amazon(Berlin),Berlin,Berlin,Germany e-mail:[email protected] S.Escalera((cid:2)) UniversitatdeBarcelonaandComputerVisionCenter,Barcelona,Spain e-mail:[email protected] ©SpringerNatureSwitzerlandAG2020 1 S.Escalera,R.Herbrich(eds.),TheNeurIPS’18Competition,TheSpringerSeries onChallengesinMachineLearning,https://doi.org/10.1007/978-3-030-29135-8_1 2 R.HerbrichandS.Escalera ing methods and were used in the early 2000’s to assess the quality of modern machine learning algorithms such as the support vector machine [7] and Gaussian processes[8]—totheNetflixcompetition3 whichsignificantlyacceleratedresearch in recommender systems, competitions have been pivotal to the progression of appliedAIscience. As competitions became not only a driving force for areas of AI research but also an efficient way to discover experts for applied data science problems, a new startup called Kaggle was founded in 2010 around the idea of making it easy for any company to organize competitions around data science. Kaggle has run hundreds of machine learning competitions since its foundation ranging from improving gesture recognition to improving the search for the Higgs boson at CERN(see,e.g.Adam-Bourdariosetal.[1]).Competitionshaveresultedinmany successfulprojects,forexample,advancingthestateoftheartinHIVresearch[6]. In addition, Kaggle competitions were pivotal in demonstrating the practicability ofdeepneuralnetworksaswellasXGBoostandrandomforest(whichwereoften thealgorithmsusedinwinningKagglecompetitionentries).Akeytocontinuously driving progression through a Kaggle competition is a live leaderboard which incentivizesparticipants. Finally, one of the key values of a competition is that they evaluate the performance of an entire system rather than a well-isolated task or component. A competition requires participants to simultaneously solve the problem of data analysis and predictive algorithm selection as well as the system aspects such as executionspeedandresourceaswellasimplementationstabilityandrecoveryfrom hardware failure. These aspects are critical when it comes to evaluation not only the feasibility of an algorithm extracting key predictive signal from a dataset but theimplementationinarunningproductionsystems.Competitionsaretheultimate litmustestforproductionreadinessofascientificbreakthrough. 2 OverviewofNeurIPS2018Competitions In 2017, for the first time in nearly 30 years, the annual Neural Information Processing Systems conference introduced the competition track which received 23 proposals related to data-driven and live competitions in the field of AI. The proposals were reviewed by researchers and experts in challenges organization. Five competitions were accepted to be run and present their results during the 3InOctober2006,Netflixprovidedatrainingdatasetof100,480,507ratings(rangingfrom1to 5stars)that480,189usersgaveto17,770movies.Thecompetitionhadaprizemoneyof$1M forimprovingtheroot-meansquareerrorofNetflix’baselineCinematchsystemby10%.ByJune 2007,over20,000teamsfromover150countrieshadregisteredforthecompetitionandofthose, 2000teamshadsubmittedover13,000predictionsets.InJune2009,ateamfromAT&TBellLabs wontheNetflixcompetition[3].