Short-term traffic prediction under normal and abnormal conditions Fangce Guo A thesis submitted for the degree of Doctor of Philosophy of Imperial College London Centre for Transport Studies Department of Civil and Environmental Engineering Imperial College London, United Kingdom July 2013 Abstract Intelligent Transport Systems (ITS) is a field that has developed rapidly over the last two decades, driven by the growing need for better transport network management strategies and by continuing improvements in computing power. However, a number of ITS applications, such as Advanced Traveller Information Systems (ATIS), Dynamic Route Guidance (DRG) and Urban Traffic Control (UTC) need to be proactive rather than reactive, and consequently require the prediction of traffic state variables into the short-term future. Similarly, individual travellers can use this predictive information to plan their mobility more efficiently. This PhD thesis develops models that are able to accurately predict short-term traffic variables such as link travel time and traffic flow on urban arterial roads under both normal and abnormal traffic conditions. This research first reviews the state of the art in data prediction applications in engineering domains especially traffic engineering and presents existing statistical and machine learning methods and their applications in relation to short-term traffic prediction. This review establishes that most existing work has focused on the apparent superiority of one individual statistical or machine learning method over another. Little attention has been paid, however, to the issues surrounding the overall structure of prediction models, in particular in relation to data smoothing and error feedback. In developing a short-term traffic prediction model, therefore, a 3-stage framework including a data smoothing step and an error feedback mechanism is Page | 2 proposed. This proposed framework is applied in conjunction with five different machine learning methods to develop a range of short-term traffic prediction methods. The proposed prediction framework is then tested under different traffic conditions using traffic data generated from a traffic simulation model of a corridor in Southampton. The prediction results show that the proposed 3-stage prediction framework can improve the accuracy of traffic prediction, regardless of the machine learning method used under both normal and abnormal traffic conditions. After demonstrating the effectiveness of predicting traffic variables using simulated data, the proposed methodology is then applied to real-world traffic data collected from different sites in London and Maidstone. These results also show that the framework can improve the accuracy of prediction regardless of the machine learning tool used. The prediction accuracy comparison shows that the proposed 3-stage prediction framework can improve the prediction accuracy for either travel time or traffic flow data under both normal and abnormal traffic conditions. In addition, the results indicate that the kNN based prediction method, when applied through the proposed framework, outperforms other selected machine learning methods under abnormal traffic conditions on urban roads. The findings suggest that, in order to arrive at a robust and accurate prediction model, attention should be paid to combining data smoothing, model structure and error feedback elements. Page | 3 Declaration of Originality At various stages during this PhD, I have been involved in collaborative efforts with both academic and industrial colleagues. In certain cases, the output of this collaboration is included in this thesis to better explain and support the research presented. In particular, my research has built upon collaborative work with my supervisors and other colleagues, working on several collaborative research papers that were presented at various conferences and submitted for journal publication. These are listed in the reference section and are all my own work. I hereby declare that besides the collaboration referred to above I have personally carried out the work described in this dissertation. ………………………. Fangce Guo Page | 4 Copyright Declaration The copyright of this thesis rests with the author and is made available under a Creative Commons Attribution Non-Commercial No Derivatives licence. Researchers are free to copy, distribute or transmit the thesis on the condition that they attribute it, that they do not use it for commercial purposes and that they do not alter, transform or build upon it. For any reuse or redistribution, researchers must make clear to others the licence terms of this work. Page | 5 Acknowledgements First and foremost I would like to thank my supervisors Professor John Polak and Dr Rajesh Krishnan for offering me the opportunity to study in Intelligent Transport Systems (ITS). Without their inspirational guidance, excellent supervision and financial support, this thesis would not have been accomplished. I am very grateful to Martin Wylie of Southampton City Council, who provided me with the AIMSUN micro-simulation model of Southampton. I would like to thank Chunkin Cheung and Andy Emmonds of Transport for London for providing me with travel time data for the A40 road in London. I must also thank John Murdoch of Kent County Council and Malcolm Kersey of Jacobs for providing the traffic data for Kent used within this thesis. I would like to thank Dr Robin North and Dr Tzu-Chang (Joe) Lee for the many useful suggestions in the early stage of my PhD research, Dr Simon Hu in explaining simulation related issues and Dr Jack Han for the many useful discussions on ITS related topics from traffic data collection to traffic estimation. I would also like to express my gratitude to my colleagues and officemates in Room 609 and 613 and to Mrs Jackie Sime for her administrative help during the past four years. Special thanks go to my friends, Siyi Li and Ada Hao in China, who are always there to encourage me via Skype and facetime when I need them. Page | 6 Last but not least, I dedicate this work to my parents and other family members in Shenyang for their continuous support and encouragement, and to my husband Hongda for his patience and sacrifices. Without your love this thesis would never have been finished. Page | 7 Contents Abstract ........................................................................................................................... 2 Declaration of Originality ............................................................................................. 4 Copyright Declaration ................................................................................................... 5 Acknowledgements ........................................................................................................ 6 Contents .......................................................................................................................... 8 List of Figures ............................................................................................................... 15 List of Tables ................................................................................................................ 21 Chapter 1 Introduction ............................................................................................ 24 1.1 Background ........................................................................................................ 25 1.1.1 Short-term traffic prediction problem statement......................................... 26 1.1.2 Factors influencing traffic conditions ......................................................... 28 1.2 Research scope and objectives ........................................................................... 29 1.2.1 Research scope ............................................................................................ 29 1.2.2 Research objectives ..................................................................................... 30 1.2.3 Research considerations .............................................................................. 30 1.2.3.1 Prediction accuracy ............................................................................... 30 1.2.3.2 Model robustness .................................................................................. 31 Page | 8 1.2.3.3 Ease of implementation and transferability .......................................... 31 1.3 Structure of this thesis ........................................................................................ 31 Chapter 2 Review of Short-term Data Prediction Methods ................................. 33 2.1 Introduction ........................................................................................................ 33 2.2 Short-term traffic prediction methods ................................................................ 33 2.3 Factors influencing short-term traffic prediction models .................................. 35 2.3.1 Implementation context for short-term traffic prediction ........................... 35 2.3.2 Input variables in short-term traffic prediction ........................................... 36 2.3.3 Input data resolution in short-term traffic prediction .................................. 38 2.3.4 Prediction steps in short-term traffic prediction ......................................... 38 2.3.5 Seasonal temporal and spatial patterns in short-term traffic prediction ..... 39 2.3.6 Traffic conditions in short-term traffic prediction ...................................... 39 2.3.6.1 Traffic prediction under normal traffic conditions ............................... 40 2.3.6.2 Traffic prediction under abnormal traffic conditions ............................ 41 2.3.7 Summary ..................................................................................................... 43 2.4 Short-term data prediction in other domains ..................................................... 43 2.4.1 Short-term data prediction in finance ......................................................... 44 2.4.2 Short-term data prediction in hydrology ..................................................... 46 2.4.3 Short-term data prediction in energy .......................................................... 47 2.4.4 Summary ..................................................................................................... 49 2.5 Review of statistical and machine learning methods in traffic prediction ......... 50 Page | 9 2.5.1 Historical average ....................................................................................... 51 2.5.2 Statistical methods ...................................................................................... 51 2.5.3 Grey System Model (GM) .......................................................................... 54 2.5.4 Kalman filter (KF) ...................................................................................... 58 2.5.5 Neural Network (NN) ................................................................................. 60 2.5.6 K-Nearest Neighbour method (kNN) .......................................................... 65 2.5.7 Kernel Smoothing (KS) .............................................................................. 70 2.5.8 Spinning Network (SPN) ............................................................................ 71 2.5.9 Support Vector Regression (SVR) .............................................................. 73 2.5.10 Random Forests (RF) ................................................................................ 76 2.6 Summary of existing traffic prediction methods ............................................... 80 2.7 Conclusions ........................................................................................................ 89 Chapter 3 Short-term Traffic Prediction Frameworks ........................................ 90 3.1 Background ........................................................................................................ 90 3.2 Data smoothing .................................................................................................. 92 3.2.1 Overview of formal data smoothing approaches ........................................ 93 3.2.2 The SSA method ......................................................................................... 94 3.2.3 Prediction framework with data smoothing ................................................ 99 3.3 Machine learning methods ............................................................................... 102 3.3.1 Introduction ............................................................................................... 102 3.3.2 kNN ........................................................................................................... 102 Page | 10