Studies in Computational Intelligence 711 Brijesh Verma Ligang Zhang David Stockwell Roadside Video Data Analysis: Deep Learning Studies in Computational Intelligence Volume 711 Series editor Janusz Kacprzyk, Polish Academy of Sciences, Warsaw, Poland e-mail: [email protected] About this Series The series “Studies in Computational Intelligence” (SCI) publishes new develop- mentsandadvancesinthevariousareasofcomputationalintelligence—quicklyand with a high quality. The intent is to cover the theory, applications, and design methods of computational intelligence, as embedded in the fields of engineering, computer science, physics and life sciences, as well as the methodologies behind them. The series contains monographs, lecture notes and edited volumes in computational intelligence spanning the areas of neural networks, connectionist systems, genetic algorithms, evolutionary computation, artificial intelligence, cellular automata, self-organizing systems, soft computing, fuzzy systems, and hybrid intelligent systems. Of particular value to both the contributors and the readership are the short publication timeframe and the worldwide distribution, which enable both wide and rapid dissemination of research output. More information about this series at http://www.springer.com/series/7092 Brijesh Verma Ligang Zhang (cid:129) David Stockwell Roadside Video Data Analysis: Deep Learning 123 Brijesh Verma DavidStockwell Schoolof Engineering andTechnology Schoolof Engineering andTechnology Central Queensland University Central Queensland University Brisbane, QLD Brisbane, QLD Australia Australia LigangZhang Schoolof Engineering andTechnology Central Queensland University Brisbane, QLD Australia ISSN 1860-949X ISSN 1860-9503 (electronic) Studies in Computational Intelligence ISBN978-981-10-4538-7 ISBN978-981-10-4539-4 (eBook) DOI 10.1007/978-981-10-4539-4 LibraryofCongressControlNumber:2017936905 ©SpringerNatureSingaporePteLtd.2017 Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar methodologynowknownorhereafterdeveloped. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authorsortheeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinor for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations. Printedonacid-freepaper ThisSpringerimprintispublishedbySpringerNature TheregisteredcompanyisSpringerNatureSingaporePteLtd. Theregisteredcompanyaddressis:152BeachRoad,#21-01/04GatewayEast,Singapore189721,Singapore Preface Video data analysis has become an increasingly important research area with widespread applications in automatic surveillance of transportation infrastructure including roads, rail and airports. As the amount of video data collected grows, so doestheopportunityforfurtherprocessingwithnewartificialintelligencemethods. Onetypeofusefulvideodatathathasseenlittleresearchistheroadsidevideodata that is collected using video-mounted vehicles. These video data may augment or replace road-based surveys of the conditions of roadside objects such as trees, grasses, roads and traffic signs and can be potentially used in many real-world applications such as roadside vegetation growth condition monitoring, effective roadside management to reduce the possible hazards to drivers and vehicles, and developingautomaticvehiclesthatareabletoautomaticallysenseroadsideobjects and traffic signs. Mostexisting studies onvideo data analysisareprimarily focusedonanalyzing generic object categories in the data content in public benchmark datasets. Very limited research has focused on the analysis of roadside video data, although the significance of developing smart techniques for roadside video data analysis has beenwidelyrecognized.Oneofthemainreasonsisprobablybecausethereisalack ofacomprehensivepublicdatasetthatwasspecificallycreatedforroadsideobjects. Another reason is the various types of variations and environmental conditions encountered along road sides, which are still challenging issues in the computer visionfield.Thegreatvariabilityintheappearanceandstructure ofobjectsaswell asthevarioustypesofenvironmentaleffectssuchasunderexposure,overexposure, shadows, and sunlight reflectance make accurate segmentation and recognition of objects difficult. The current literature lacks a comprehensive review of existing machine learning algorithms, particularly deep learning techniques, on roadside data analysis. This book highlights the methods and applications for roadside video data analysis. It describes various system architectures and methodologies that are specificallybuiltupondifferenttypesoflearningalgorithmsforroadsidevideodata processing, with detailed analysis of the segmentation, feature extraction and classification. The use of deep learning to solve the roadside video data v vi Preface segmentation and classification problems is one of the major highlights of this book. Deep neural net learning has become popular in machine learning and data mining areas. However, the benefits of a deep feature free approach must be bal- anced against the considerations of accuracy and robustness, and most real-world learningsystems requiresome hand engineering offeatures andarchitectures. This book examines via empirical testing the types of features and architectures that contribute to the performance of multi-layer neural nets on real-world scene anal- ysis.Wethendemonstratenovelarchitecturesthatperformsceneclassificationwith equalorbetteraccuracytopreviousmethodsandinvestigatethefeatureengineering intoconvolutionalneuralnetworks.Further,weprovideanindustrialperspectiveto help align theoretical concerns with real-world results. Finally, as a case study of roadside video data analysis, we demonstrate an application of vegetation biomass estimationtechniquesforroadsidefireriskassessment.Overall,thisbookcompiles the most useful strategies in the field of scene analysis to help researchers identify the most appropriate features and architectures for their applications. Brisbane, Australia Brijesh Verma Ligang Zhang David Stockwell Acknowledgements The authors express their gratitude for help from the Department of Transport and Main Roads in Queensland, Australia for the creation of the data and assistance with field resources for the fire risk survey. This work was supported under Australian Research Council’s Linkage Projects funding scheme (project number LP140100939). Many students and research fellows in the Centre for Intelligent Systems at CentralQueenslandUniversityhavecontributedtoresearchpresentedinthisbook. The authors would like to thank the following researchers: Dr Sujan Chowdhury, DrTejyKinattukaraJobachan,DrPeterMcLeod,DrM.Asafuddoula,Mrs.Toshi Sinha and Mrs. Fatma Shaheen. vii Contents 1 Introduction . .... .... .... ..... .... .... .... .... .... ..... .. 1 1.1 Background .. .... .... ..... .... .... .... .... .... ..... .. 1 1.2 Collection of Roadside Video Data . .... .... .... .... ..... .. 2 1.2.1 Industry Data ... ..... .... .... .... .... .... ..... .. 2 1.2.2 Benchmark Data. ..... .... .... .... .... .... ..... .. 7 1.3 Applications Using Roadside Video Data .... .... .... ..... .. 10 1.4 Outline of the Book.... ..... .... .... .... .... .... ..... .. 11 References ... .... .... .... ..... .... .... .... .... .... ..... .. 12 2 Roadside Video Data Analysis Framework.. .... .... .... ..... .. 13 2.1 Overview.... .... .... ..... .... .... .... .... .... ..... .. 13 2.2 Methodology . .... .... ..... .... .... .... .... .... ..... .. 14 2.2.1 Pre-processing of Roadside Video Data.... .... ..... .. 14 2.2.2 Segmentation of Roadside Video Data into Objects.... .. 18 2.2.3 Feature Extraction from Objects.. .... .... .... ..... .. 19 2.2.4 Classification of Roadside Objects.... .... .... ..... .. 20 2.2.5 Applications of Classified Roadside Objects. .... ..... .. 21 2.3 Related Work. .... .... ..... .... .... .... .... .... ..... .. 23 2.3.1 Vegetation Segmentation and Classification. .... ..... .. 24 2.3.2 Generic Object Segmentation and Classification.. ..... .. 28 2.4 Matlab Code for Data Processing... .... .... .... .... ..... .. 32 References ... .... .... .... ..... .... .... .... .... .... ..... .. 37 3 Non-deep Learning Techniques for Roadside Video Data Analysis. .... .... .... .... ..... .... .... .... .... .... ..... .. 41 3.1 Neural Network Learning..... .... .... .... .... .... ..... .. 41 3.1.1 Introduction .... ..... .... .... .... .... .... ..... .. 41 3.1.2 Neural Network Learning Approach... .... .... ..... .. 42 3.1.3 Experimental Results .. .... .... .... .... .... ..... .. 45 3.1.4 Summary .. .... ..... .... .... .... .... .... ..... .. 48 ix x Contents 3.2 Support Vector Machine Learning.. .... .... .... .... ..... .. 49 3.2.1 Introduction .... ..... .... .... .... .... .... ..... .. 49 3.2.2 SVM Learning Approach... .... .... .... .... ..... .. 49 3.2.3 Experimental Results .. .... .... .... .... .... ..... .. 53 3.2.4 Summary .. .... ..... .... .... .... .... .... ..... .. 56 3.3 Clustering Learning.... ..... .... .... .... .... .... ..... .. 56 3.3.1 Introduction .... ..... .... .... .... .... .... ..... .. 56 3.3.2 Clustering Learning Approach ... .... .... .... ..... .. 57 3.3.3 Experimental Results .. .... .... .... .... .... ..... .. 62 3.3.4 Summary .. .... ..... .... .... .... .... .... ..... .. 73 3.4 Fuzzy C-Means Learning..... .... .... .... .... .... ..... .. 74 3.4.1 Introduction .... ..... .... .... .... .... .... ..... .. 74 3.4.2 Fuzzy C-Means Learning Approach... .... .... ..... .. 75 3.4.3 Experimental Results .. .... .... .... .... .... ..... .. 78 3.4.4 Summary .. .... ..... .... .... .... .... .... ..... .. 80 3.5 Ensemble Learning .... ..... .... .... .... .... .... ..... .. 80 3.5.1 Introduction .... ..... .... .... .... .... .... ..... .. 80 3.5.2 Ensemble Learning Approach.... .... .... .... ..... .. 80 3.5.3 Experimental Results .. .... .... .... .... .... ..... .. 84 3.5.4 Summary .. .... ..... .... .... .... .... .... ..... .. 86 3.6 Majority Voting Based Hybrid Learning . .... .... .... ..... .. 86 3.6.1 Introduction .... ..... .... .... .... .... .... ..... .. 86 3.6.2 Majority Voting Approach.. .... .... .... .... ..... .. 87 3.6.3 Experimental Results .. .... .... .... .... .... ..... .. 90 3.6.4 Summary .. .... ..... .... .... .... .... .... ..... .. 94 3.7 Region Merging Learning .... .... .... .... .... .... ..... .. 95 3.7.1 Introduction .... ..... .... .... .... .... .... ..... .. 95 3.7.2 Region Merging Approach.. .... .... .... .... ..... .. 95 3.7.3 Components of Approach... .... .... .... .... ..... .. 100 3.7.4 Experimental Results .. .... .... .... .... .... ..... .. 109 3.7.5 Summary .. .... ..... .... .... .... .... .... ..... .. 115 References ... .... .... .... ..... .... .... .... .... .... ..... .. 116 4 Deep Learning Techniques for Roadside Video Data Analysis... .. 119 4.1 Introduction.. .... .... ..... .... .... .... .... .... ..... .. 119 4.2 Related Work. .... .... ..... .... .... .... .... .... ..... .. 120 4.3 Automatic Versus Manual Feature Extraction . .... .... ..... .. 122 4.3.1 Introduction .... ..... .... .... .... .... .... ..... .. 122 4.3.2 Comparison Framework.... .... .... .... .... ..... .. 122 4.3.3 Experimental Results .. .... .... .... .... .... ..... .. 124 4.3.4 Summary .. .... ..... .... .... .... .... .... ..... .. 125 4.4 Single Versus Ensemble Architectures... .... .... .... ..... .. 127 4.4.1 Introduction .... ..... .... .... .... .... .... ..... .. 127 4.4.2 Comparison Framework.... .... .... .... .... ..... .. 128