Deep Learning for Time-Series Analysis John Gamboa University of Kaiserslautern Kaiserslautern, Germany Abstract. In many real-world application, e.g., speech recognition or sleep stage classification, data are captured over the course of time, constituting a Time-Series. Time-Series often contain temporal depen- dencies that cause two otherwise identical points of time to belong to 7 different classes or predict different behavior. This characteristic gener- 1 ally increases the difficulty of analysing them. Existing techniques often 0 dependedonhand-craftedfeaturesthatwereexpensivetocreateandre- 2 quired expert knowledge of the field. With the advent of Deep Learning n newmodelsofunsupervisedlearningoffeaturesforTime-seriesanalysis a andforecasthavebeendeveloped.Suchnewdevelopmentsarethetopic J of this paper: a review of the main Deep Learning techniques is pre- 7 sented, and some applications on Time-Series analysis are summaried. The results make it clear that Deep Learning has a lot to contribute to ] G the field. L Keywords: Artificial Neural Networks, Deep Learning, Time-Series . s c [ 1 Introduction 1 v Artificial Neural Networks (ANN), since their origin in 1943 [46], have been 7 used to solve a large range of problems as diverse as robotic processing [32], 8 8 object recognition [56], speech and handwriting recognition [19], and even real 1 timesign-languagetranslation[2].Despitetheintuitionthatdeeperarchitectures 0 would yield better results than the then more commonly used shallow ones, . 1 empiricaltestswithdeepnetworkshadfoundsimilarorevenworseresultswhen 0 compared to networks with only one or two layers [57] (for more details, see 7 [6]). Additionally, training was found to be difficult and often inefficient [6]. 1 L¨angkvist [34] argues that this scenario started to change with the proposal of : v greedy layer-wise unsupervised learning [22], which allowed for the fast learning Xi of Deep Belief Networks, while also solving the vanishing gradients problem [7]. Latest deep architectures use several modules that are trained separately and r a stacked together so that the output of the first one is the input of the next one. Fromstockmarketpricestothespreadofanepidemic,andfromtherecord- ing of an audio signal to sleep monitoring, it is common for real world data to be registered taking into account some notion of time. When collected together, themeasurementscomposewhatisknownasaTime-Series.Fordifferentfields, suitable applications vary depending on the nature and purpose of the data: 2 Deep Learning for Time-Series Analysis while doctors can be interested in searching for anomalies in the sleep patterns of a patient, economists may be more interested in forecasting the next prices somestocksofinterestwillassume.Thesekindsofproblemsareaddressedinthe literaturebyarangeofdifferentapproches(forarecentreviewofthemaintech- niques applied to perform tasks such as Classification, Segmentation, Anomaly Detection and Prediction, see [14]). This paper reviews some of the recently presented approaches to performing tasks related to Time-Series using Deep Learning architectures. It is important, therefore,tohaveaformaldefinitionofTime-Series.Malhotraetal.[44]defined Time-SeriesasavectorX ={x(1),x(2),...,x(n)},whereeachelementx(t) ∈Rm pertaining to X is an array of m values such that {x(t),x(t),...,x(t)}. Each one 1 2 m of the m values correspond to the input variables measured in the time-series. The rest of this paper is structured as follows: Section 1.1 introduces basic types of Neural Network (NN) modules that are often used to build deep neural structures. Section 2 describes how the present paper relates to other works in theliterature.Sections3,4and5describesomeapproachesusingDeepLearning toperformModeling,ClassificationandAnomalyDetectioninTime-Seriesdata, respectively. Finally, Section 6 concludes the paper. 1.1 Artificial Neural Network This section explains the basic concepts related to ANN. The types of networks describedherearebynomeanstheonlykindsofANNarchitecturesfoundinthe literature.Thereaderisreferredto[49]forathoroughdescriptionofarchitectural alternatives such as Restricted Boltzmann Machines (RBM), Hopfield Networks andAuto-Encoders,aswellasforadetailedexplanationoftheBackpropagation algorithm. Additionally, we refer the reader to [19] for applications of RNN as well as more details on the implementation of a LSTM, and to [54] for details on CNN. AnANNisbasicallyanetworkofcomputingunitslinkedbydirectedconnec- tions. Each computing unit performs some calculation and outputs a value that is then spread through all its outgoing connections as input into other units. Connectionsnormallyhaveweightsthatcorrespondtohowstrongtwounitsare linked. Typically, the computation performed by a unit is separated into two stages: the aggregation and the activation functions. Applying the aggregation functioncommonlycorrespondstocalculatingthesumoftheinputsreceivedby the unit through all its incoming connections. The resulting value is then fed into the activation function. It commonly varies in different network architec- tures, although popular choices are the logistic sigmoid (σ(x)= 1 ) and the 1+e−x hyperbolic tangent (tanh(x) = 2 −1) functions. Recently, rectified linear 1+e−2x units employing a ramp function (R(x) = max(0,x)) have become increasingly popular. The input of the network is given in a set of input computing units which compose an input layer. Conversely, the output of the network are the values outputbytheunitscomposingtheoutputlayer.Allotherunitsarecalledhidden and are often also organized in layers (see Figure 1b for an example network). Deep Learning for Time-Series Analysis 3 output f Output Gate h Forget Gate Cell f ... hidden ... i i i 1 2 3 f Input Gate w v z w v z w v z g Block n11 n12 n13... n21 n22 n23... n31 n32 n33 input (b) (a) (c) Fig.1: (a) The convolutional layer of a CNN with three groups (also called “fil- ters”). Each group performs a 2×2 convolution in the image: each neuron in the group is connected to a different region of the image but shares the same weights, producing a new image. In the example, the weight vector w is shared by all neurons n , the vector v is shared by all neurons n , and z is shared j1 j2 by all neurons n . If pooling is applied, it is applied to each one of the three j3 newly produced images; (b) An Artificial Neural Network with one input layer composed of three neurons, two hidden layers composed, each one, of four neu- rons, and one output layer composed of three neurons. Each node of a layer is connected to all nodes of the next layer; (c) A LSTM block (adapted from [19]). Input layer LSTM/Sigmoid H1 + G1 G0 Layer H2 + G2 LSTM/Sigmoid H G Layer 3 3 Output layer (b) (a) Fig.2: (a) The proposed “Stacked Architecture” for performing Anomaly De- tection (adapted from [44]); (b) The architecture of a UFCNN (adapted from [47]). 4 Deep Learning for Time-Series Analysis Thefocusoflearningalgorithmsisfrequentlyondecidingwhatweightswould cause the network to output, given some input, the expected values. A popular learning algorithm is the Backpropagation algorithm [51], whereby the gradient of an error function is calculated and the weights are iteratively set so as to minimize the error. Convolutional Neural Network (CNN) Anetworkthatistoobigandwith layers that are fully connected can become infeasible to train. Also trained with the Backpropagation algorithm, CNNs [36] are common for image processing tasksandreducethenumberofparameterstobelearnedbylimitingthenumber of connections of the neurons in the hidden layer to only some of the input neurons (i.e., a local area of the input image). A hidden layer (in this case, also called a convolutional layer – see Figure 1a) is composed by several groups of neurons.Theweightsofallneuronsinagroupareshared.Eachgroupisgenerally composed by as many neurons as needed to cover the entire image. This way, it is as if each group of neurons in the hidden layer calculated a convolution of the image with their weights, resulting in a “processed” version of the image. We call this convolution a feature. Commonly, pooling is applied to the resulting filtered images. The tecnique allows for achieving some translation invariance of the learned features. The groups (containing a newly processed version of the input image) are divided in chunks (e.g., 2×2) and their maximum value is taken. This results in yet another version of the input image, now smaller than the original size (in the example, 1/4 of the size of the group). These steps can be repeatedly applied as many times as desired: a new con- volutional layer can be applied on the pooled layer, followed by another pooling layer, and so forth. Finally, when the layers become small enough, it is common to have fully connected layers before the output layer. Tiled Convolutional Neural Network The usage of shared weights in a group allowforthetranslationinvarianceofthelearnedfeatures.However,“itprevents the pooling units from capturing more complex invariances, such as scale and rotationinvariance”[48].Tosolvethisproblem,TiledCNNsallowforagroupto be divided into subgroups called tiles, each of which can have separate weights. A parameter k defines how many tiles each group has: neurons that are exactly k steps away from each other share the same weights. Fully Convolutional Networks (FCN) Whilethepoolingoperationperformedby CNNs makes sense for object recognition tasks, because it has the advantage of achieving some robustness to small shifts of the learned features, it is not suited for tasks like Semantic Segmentation, where the goal is to segment the pixels of the image according to the objects that they refer to. FCNs [42] allow for the input and output layers to have the same dimensions by introducing “a decoder stage that is consisted of upsampling, convolution, and rectified linear units layers, to the CNN architecture” [47]. Deep Learning for Time-Series Analysis 5 RecurrentNeuralNetwork(RNN) Whenthenetworkhasloops,itiscalled aRNN.ItispossibletoadapttheBackpropagationalgorithmtotrainarecurrent network,by“unfolding”thenetworkthroughtimeandconstrainingsomeofthe connections to always hold the same weights [51]. Long Short-Term Memory (LSTM) One problem that arises from the unfolding ofanRNNisthatthegradientofsomeoftheweightsstartstobecometoosmall or too large if the network is unfolded for too many time steps. This is called the vanishing gradients problem [7]. A type of network architecture that solves this problem is the LSTM [23]. In a typical implementation, the hidden layer is replaced by a complex block (see Figure 1c) of computing units composed by gates that trap the error in the block, forming a so-called “error carrousel”. 2 Literature Review Independently of Deep Learning, analysis of Time-Series data have been a pop- ular subject of interest in other fields such as Economics, Engineering and Medicine. Traditional techniques on manipulating such data can be found in [21], and the application of traditional ANN techniques on this kind of data is described in [5]. Most work using ANN to manipulate Time-Series data focuses on modeling and forecasting. As an early attempt on using ANN for such tasks, [9] modelled flour prices over the range of 8 years. Still in the 90’s, [29] delineated eight steps on“designinganeuralnetworkforecastmodelusingeconomictimeseriesdata”. More recent approaches include usage of Elman RNNs to predict chaotic Time- Series [10], employing ANN ensemble methods for forecasting Internet traffic [12], using simple Multilayer Perceptrons for modeling the amount of littering in the North Sea [53], and implementing in FPGA a prediction algorithm using Echo State Networks for “exploiting the inherent parallelism of these systems” [3]. Hybrid approaches to Time-Series analysis utilizing ANN are not uncom- mon. [31] presents a model for Time-Series forecasting using ANN and ARIMA models, and [15] applies the same kinds of models to water quality time series prediction. In still other examples of the same ideas, [30] compares the perfor- mance of ARIMA models and ANNs to make short-term predictions on photo- voltaicpowergenerators,while[38]comparesbothmodelswiththeperformance ofMultivariateAdaptiveRegressionSplines.[13]performsTime-Seriesforecast- ing by using a hybrid fuzzy model: while the Fuzzy C-means method is utilized for fuzzification, ANN are employed for defuzzification. Finally, [24] forecasts the speed of the wind using a hybrid of Support Vector Machines, Ensemble Empirical Mode Decomposition and Partial Autocorrelation Function. Despite being relatively new, the field of Deep Learning has attracted a lot of interest in the past few years. A very thorough review of the entire history of developments that led the field to its current state can be found in [52], while a higher focus on the novelties from the last decade is given in [4]. We proceed to a review of the applications of Deep Learning to Time-Series data. 6 Deep Learning for Time-Series Analysis Classification The task of Classification of any type of data has benefited by the advent of CNNs. Previously existing methods for classification generally relied on the usage of domain specific features normally crafted manually by human experts. Finding the best features was the subject of a lot of research andtheperformanceoftheclassifierwasheavilydependentontheirquality.The advantage of CNNs is that they can learn such features by themselves, reducing the need for human experts [34]. An example of the application of such unsupervised feature learning for the classification of audio signals is presented in [37]. In [1], the features learned by the CNN are used as input to a Hidden Markov Model, achieving a drop at the error rate of over 10%. The application of CNNs in these works presuppose the constraintthattheTime-Seriesiscomposedofonlyonechannel.Anarchitecture that solves this constraint is presented in [60]. In [18] the performance of CNNs is compared with that of LSTM for the classification of Visual and Haptic Data in a robotics setting, and in [28] the signals produced by wearable sensors are transformed into images so that Deep CNNs can be used for classification. Relevant to Tiled CNNs was the development of Independent Component Analysis (ICA) [27]. Several alternative methods for calculating independent componentscanbefoundintheliterature(e.g.,[17],[55]or[25]).TiledCNNsare normally trained with a variation of such technique that looses the assumption that each component is statistically independent and tries to find a topographic order between them: the Topographic ICA [26]. Forecasting Several different Deep Learning approaches can be found in the literature for performing Forecasting tasks. For example, Deep Belief Networks areusedintheworkof[33]alongwithRBM.[58]alsocomparestheperformance ofDeepBeliefNetworkswiththatofStackedDenoisingAutoencoders.Thislast typeofnetworkisalsoemployedby[50]topredictthetemperatureofanindoor environment. Another application of Time-Series forecasting can be found in [43], which uses Stacked Autoencoders to predict the flow of traffic from a Big Data dataset. A popular application to the task of Time-Series prediction is on Weather Forecasting. In [41], some preliminary predictions on weather data provided by The Hong Kong Observatory are made through the usage of Stacked Autoen- coders.Inafollowupwork,theauthorsusesimilarideastoperformpredictions on Big Data [40]. Instead of Autoencoders, [20] uses Deep Belief Networks for constructing a hybrid model in which the ANN models the joint distribution between the weather predictors variables. Anomaly Detection Work applying Deep Learning techniques to Anomaly Detection detection of Time-Series data is not very abundant in the literature. It is still difficult to find works such as [16], that uses Stacked Denoising Au- toencoderstoperformAnomalyDetectionoftrajectoriesobtainedfromlowlevel tracking algorithms. Deep Learning for Time-Series Analysis 7 However, there are many similarities between Anomaly Detection and the previous two tasks. For example, identifying an anomaly could be transformed intoaClassificationtask,aswasdonein[35].Alternatively,detectingananomaly couldbeconsideredthesameasfindingregionsintheTime-Seriesforwhichthe forecasted values are too different from the actual ones. 3 Deep Learning for Time-Series Modeling In this section the work presented in [47] is reviewed. As discussed above, FCNs are a modification of the CNN architecture that, as required by some Time- Series related problems, allows for the input and output signals to have the same dimensions. Mittelman [47] argues that the architecture of the FCN resembles the ap- plication of a wavelet transform, and that for this reason, it can present strong variations when the input signal is subject to small translations. To solve this problem, and inspired by the undecimeated wavelet transform, which is transla- tion invariant, they propose the Undecimated Fully Convolutional Neural Net- work (UFCNN), also translation invariant. The only difference between an FCN and an UFCNN is that the UFCNN removesboththeupsamplingandpoolingoperatorsfromtheFCNarchitecture. Instead, the “filters at the lth resolution level are upsampled by a factor of 2l−1 along the time dimension”. See Figure 2b for a graphical representation of the proposed architecture. The performance of the UFCNN is tested in three different experiments. In the first experiment, “2000 training sequences, 50 validation sequences and 50 testingsequences,eachoflength5000time-steps”areautomaticallygeneratedby aprobabilisticalgorithm.ThevaluesoftheTime-Seriesrepresentthepositionof atargetobjectmovinginsideaboundedsquare.TheperformanceoftheUFCNN in estimating the position of the object at each time-step is compared to that of a FCN, a LSTM, and a RNN, and the UFCNN does perform better in most cases. In a second experiment, the “MUSE” and “NOTTINGHAM” datasets1 area used. The goal is to predict the values of the Time-Series in the next time- step. In both cases, the UFCNN outperforms the competing networks: a RNN, a Hessian-Free optimization-RNN [45], and a LSTM. Finally, the third experiment uses a trading dataset2, where the goal is, given only information about the past, to predict the set of actions that would “maximize the profit”. In a comparison to a RNN, the UFCNN again yielded the best results. 1 available at http://www-etud.iro.umontreal.ca/ boulanni/icml2012 2 available at http://www.circulumvite.com/home/trading-competition 8 Deep Learning for Time-Series Analysis 4 Deep Learning for Time-Series Classification WangandOates[59]presentedanapproachforTime-SeriesClassificationusing CNN-like networks. In order to benefit from the high accuracy that CNNs have achievedinthepastfewyearsonimageclassificationtasks,theauthorspropose the idea of transforming a Time-Series into an image. Two approaches are presented. The first one generates a Gramian Angular Field (GAF), while the second generates a Markov Transition Field (MTF). In both cases, the generation of the images increases the size of the Time-Series, makingtheimagespotentiallyprohibitivelylarge.Theauthorsthereforepropose strategiestoreducetheirsizewithoutloosingtoomuchinformation.Finally,the two types of images are combined in a two-channel image that is then used to producebetterresultsthanthoseachievedwhenusingeachimageseparately.In the next sections, GAF and MTF are described. Intheequationsbelow,wesupposethatm=1.TheTime-Seriesistherefore composedbyonlyreal-valuedobservations,suchthatreferringtox(i) ∈X isthe same as referring to x(i) ∈X. 4.1 Gramian Angular Field The first step on generating a GAF is to rescale the entire Time-Series into values between [−1,1]. In the equation 1, max(X) and min(X) represent the maximum and minimum real-values present in the Time-Series X: (x(i)−max(X))+(x(i)−max(X)) x˜(i) = (1) max(X)−min(X) The next step is to recode the newly created Time-Series X˜ into polar coor- dinates. The angle is encoded by x(i) and the radius is encoded by the the time stamp i. Notice that, because the values x(i) were rescaled, no information is lost by the usage of arccos(x˜(i)) in 2. (cid:40) φ=arccos(x˜(i)), −1≤x˜(i) ≤1,x˜(i) ∈X˜ (2) r = i , i∈N N Finally, the GAF is defined as follows:   cos(φ +φ ) ··· cos(φ +φ ) 1 1 1 n cos(φ2+φ1) ··· cos(φ2+φn) G= ... ... ...  (3) cos(φ +φ )··· cos(φ +φ ) n 1 n n Here, some information is lost by the fact that φ no more belongs to the interval [0,π]. When trying to recover the Time-Series from the image, there may be some errors introduced. Deep Learning for Time-Series Analysis 9 4.2 Markov Transition Field The creation of the Markov Transition Field is based on the ideas proposed in [8] for the definition of the so-called Markov Transition Matrix (MTM). For a Time-Series X, the first step is defining Q quantile bins. Each x(i) is then assigned to the corresponding bin q . The Markov Transition Matrix is the j (cid:80) matrix W composed by elements w such that w =1 and w corresponds ij j ij ij to the normalized “frequency with which a point in the quantile q is followed j by a point in the quantile q .” This is a Q×Q matrix. i The MTF is the n×n matrix M. Each pixel of M contains a value from W. The value in the pixel ij is the likelihood (as calculated when constructing W) of going from the bin in which the pixel i is to the bin in which the pixel j is:   w ··· w ij|x1∈qi,x1∈qj ij|x1∈qi,xn∈qj M =wij|x2∈...qi,x1∈qj ·.·.·. wij|x2∈...qi,xn∈qj (4) w ··· w ij|xn∈qi,x1∈qj ij|xn∈qi,xn∈qj 4.3 Performing Classification with the Generated Images The authors use Tiled CNNs to perform classifications using the images. In the reported experiments, both methods are assessed separately in 12 “hard” datasets“onwhichtheclassificationerrorrateisabove0.1withthestate-of-the- artSAX-BoPapproach”[39],whichare50Words,Adiac,Beef,Coffee,ECG200, Face (all), Lightning-2, Lightning-7, OliveOil, OSU Leaf, Swedish Leaf and Yoga [11]. The authors then suggest the usage of both methods as “colors” of the images. The performance of the resulting classifier is competitive against many of the state-of-the-art classifiers, which are also reported by the authors. 5 Deep Learning for Time-Series Anomaly Detection Anomaly Detection can be easily transformed into a task where the goal is to model the Time-Series and, given this model, find regions where the predicted values are too different from the actual ones (or, in other words, where the probability of the observed region is too low). This is the idea implemented by the paper reviewed in this section [44]. Through the learned model, not all m input variables need to be predicted. The learned model predicts, at any given time-step, l vectors with d input vari- ables, where 1≤d≤m. ThemodelingoftheTime-SeriesisdonethroughtheapplicationofaStacked LSTM architecture. The network has m input neurons (one for each input vari- able) and d × l output neurons (one neuron for each one of the d predicted variables of the l vectors that are predicted at a time-step). The hidden layers are composed by LSTM units that are “fully connected through recurrent con- nections”. Additionally, any hidden unit is fully connected to all units in the hidden layer above it. Figure 2a sketches the proposed architecture. 10 Deep Learning for Time-Series Analysis For each one of the predictions and each one of the d predicted variables, a prediction error is calculated. The errors are then used to fit a multivariate Guassiandistribution,andaprobabilityp(t) isassignedtoeachobservation.Any observation whose probability p(t) <τ is treated as an anomaly. The approach was tested in four real-world datasets. One of them, called Multi-sensor engine data is not publicly available. The other three datasets (Electrocardiograms (ECGs), Space Suttle Marotta valve time series, and Power demand dataset) are available for download3. The results demonstrated a sig- nificant improvement in capturing long-term dependencies when compared to simpler RNN-based implementations. 6 Conclusion When applying Deep Learning, one seeks to stack several independent neural network layers that, working together, produce better results than the already existing shallow structures. In this paper, we have reviewed some of these mod- ules, as well the recent work that has been done by using them, found in the literature.Additionally,wehavediscussedsomeofthemaintasksnormallyper- formed when manipulating Time-Series data using deep neural network struc- tures. Finally, a more specific focus was given on one work performing each one of these tasks. Employing Deep Learning to Time-Series analysis has yielded results in these cases that are better than the previously existing techniques, which is an evidence that this is a promising field for improvement. Acknowledgments. I would like to thank Ahmed Sheraz and Mohsin Munir for their guidance and contribution to this paper. References 1. Abdel-Hamid,O.,Mohamed,A.r.,Jiang,H.,Penn,G.:Applyingconvolutionalneu- ral networks concepts to hybrid nn-hmm model for speech recognition. In: Acous- tics,SpeechandSignalProcessing(ICASSP),2012IEEEInternationalConference on. pp. 4277–4280. IEEE (2012) 2. Akmeliawati, R., Ooi, M.P.L., Kuang, Y.C.: Real-time malaysian sign language translation using colour segmentation and neural network. In: Instrumentation and Measurement Technology Conference Proceedings, 2007. IMTC 2007. IEEE. pp. 1–6. IEEE (2007) 3. Alomar,M.L.,Canals,V.,Perez-Mora,N.,Mart´ınez-Moll,V.,Rossello´,J.L.:Fpga- based stochastic echo state networks for time-series forecasting. Computational Intelligence and Neuroscience 501, 537267 (2015) 4. Arel,I.,Rose,D.C.,Karnowski,T.P.:Deepmachinelearning-anewfrontierinarti- ficialintelligenceresearch[researchfrontier].ComputationalIntelligenceMagazine, IEEE 5(4), 13–18 (2010) 3 available at http://www.cs.ucr.edu/ eamonn/discords/