Time Series Forecasting

By

Surname, First Name

Course Name and I.D. Number

Instructor’s Name

August, 2013

1.0 Introduction

Forecasting is the process of predicting events whose actual outcomes are yet to be known. It is an important tool that helps management of various businesses or organizations to develop meaningful plans and reduce uncertainty of events in the future. To cope with the uncertainty of the future, forecasting relies on the analysis of trends obtained from past and present data. It starts with certain assumptions which are based on management’s experience, knowledge, and judgment. These estimates or assumptions are then projected into the future using a variety of powerful techniques. One example of such techniques is the time series forecasting technique.

Time series forecasting, which is the most common technique of forecasting, is used to identify specific patterns in data and to use the patterns to project future forecasts. According to Zhang (2003), it is an important area of forecasting in which past observations of data of the same variable are collected and analyzed to develop a model describing the underlying relationship. The model is then used to extrapolate the time series into the future. Successful time series forecasting depends on an appropriate model fitting. Due to the indispensable importance of the technique in numerous practical fields such as business, economics, finance, operations management, science and engineering, etc., many models have been evolved in literature. These models have all been applied to improve time series forecasting accuracy.

The main aim of this paper is to present the three widely used time series forecasting models (stochastic models, neural network models, and support vector machine (SVM) based models). The specific objectives of the paper are to give: the basic concepts of time series modeling; the application of the time series models and outcomes, and available new learning.

2.0 The Basic Concepts of Time Series Modeling

2.1 Time Series Definition

According to Adhikari and Agrawal (2013), time series is a sequential set of data points, measured typically over successive time. Mathematically, it is defined as a set of vectors x(t), t = 0, 1, 2, 3, …. where t represents the time elapsed. The variable x(t) is treated as a random variable. The measurements taken during an event in a time series are arranged in a proper chronological order (Adhikari and Agrawal, 2013).

Time series can be univariate or multivariate in nature. Time series is univariate when it contains records of a single variable. It is termed multivariate when it contains more than one variable. It can also be discrete or continuous in nature. A discrete time series is one which contains observations measured at discrete points of time. Examples of discrete time series are: population of a particular city, production of a company, exchange rates between two different currencies etc (Adhikari and Agrawal, 2013). Continuous time series are time series in which observations are measured at every instance of time. Examples of continuous time series are: temperature readings, flow of a river, concentration of a chemical process etc (Adhikari and Agrawal, 2013).

2.2. Time Series Components

From any observed data, time series are affected by four main components, namely: trend, seasonal variation, cyclical variation, and irregular components. Trend gives the long term movement in a time series. It is the ability or tendency of a time series to increase, decrease or stagnate over a long period of time. Seasonal variation has to do with the fluctuations that occur in the time series within a year. The factors that can cause seasonal variations are: climate and weather conditions, customs, traditional habits etc (Adhikari and Agrawal, 2013). It is an important factor for businessmen, shopkeepers and producers for making proper future plans. Cyclical variation describes the medium-term changes that occur in time series, caused by circumstances, which repeat in cycles. Irregular variations are caused by unpredictable influences, which are not regular and also do not repeat in a particular pattern.

2.3 Time Series Analysis

Time series forecasting is achieved by fitting a suitable model to a given time series, from which the corresponding parameters of the model are estimated by using known data values. According to Hipel and McLeod (1994), the procedure of fitting a time series to a proper model is known as Time Series Analysis. It comprises methods that attempt to understand the nature of the series and is often useful for future forecasting and simulation.

2.4 Model Parsimony

The principle of parsimony, according to Hipel and McLeod (1994); Chatfield (1996); Zhang (2003); Zhang (2007), must be considered in order to build a proper time series model. According to the principle of parsimony, the model with smallest possible number of parameters is always selected in order to provide adequate representation of the underlying time series data. Out of a number of suitable models, one is expected to consider the simplest one, still maintaining an accurate description of inherent properties of the time series. Thus the principle of model parsimony must be considered when developing models for time series forecasting.

3.0 Time Series Forecasting Models

Time series forecasting models are classified into three widely used model classes, namely: the stochastic models, neural networks, and the Vapnik’s support vector machine (SVM) based models. Concise description of application of these models in time series forecasting are given below.

3.1 Time Series Forecasting Using Stochastic Models

Time series stochastic models used for forecasting can be linear or non-linear in nature. The most popular and frequently used stochastic linear time series model is the Autoregressive Integrated Moving Average (ARIMA) model. According to Zhang (2007), the basic assumption made to implement ARIMA model is that the considered time series is linear and follows a particular known statistical distribution, such as the normal distribution. The Model has subclasses of models, such as the Autoregressive (AR), Moving Average (MA) and Autoregressive Moving Average (ARMA) models. According to Zhang (2007), for seasonal time series forecasting, Box and Jenkins had proposed a quite successful variation of ARIMA model, which is known as the Seasonal ARIMA (SARIMA). ARIMA model is popular because of its flexibility to represent several varieties of time series with simplicity as well as the associated Box-Jenkins methodology for optimal model building process.

The severe limitation of the models is the pre-assumed linear form of the associated time series which becomes inadequate in many practical situations. To overcome this drawback, various non-linear stochastic models, according to Parrelli (2001), have been proposed; however, from implementation point of view these are not so straight-forward and simple as the ARIMA models.

The non-linear stochastic models available in literature are: the famous Autoregressive Conditional Heteroskedasticity (ARCH) model and its variations like Generalized ARCH (GARCH), Exponential Generalized ARCH (EGARCH), the Threshold Autoregressive (TAR) model, the Non-linear Autoregressive (NAR) model, the Non-linear Moving Average model etc (Parrelli, 2001). These non-linear stochastic models were developed to fit to practical time series data that show non-linear patterns. According to Campbell, Lo and McKinley (1997), these non-linear stochastic time series models are divided into two branches: one includes models non-linear in mean and the other includes models non-linear in variance (heteroskedastic).

3.2 Time Series Forecasting Using Artificial Neural Networks

Artificial neural networks (ANNs) are one of the most important types of nonparameteric nonlinear models used for time series forecasting. The basic structure of and operations performed by the ANN, according to Bijari and Khashei (2011), emulate those found in a biological neural systems. The excellent feature of ANNs in time series forecasting problems is their inherent capability of non-linear modeling, without any presumption about the statistical distribution followed by the observations (Adhikari and Agrawal, 2013). The appropriate model is adaptively formed based on the given data and thus ANNs are data-driven and self-adaptive by nature (Zhang, 2003).

The most common ANN forecasting models are the multi-layer perceptrons (MLPs), which according to Zhang, Patuwo and Hu (1998) are characterized by a single hidden layer Feed Forward Network (FNN). The model is characterized by a network of three layers: the input, hidden and outer layer. These layers are connected by acyclic links. There may be more than one hidden layer and the nodes in various layers are also known as processing elements. The three-layer feed forward architecture of ANN models can be diagrammatically depicted as shown Fig. 1. FNN has another widely used variation which is known as Time Lagged Neural Network (TLNN). In TLNN, the input nodes are the time series values at some particular lags. For example, a typical TLNN for a time series can contain the input nodes as the lagged values at time t – 1 and t – 2. The value at time t is to be forecasted using the values at lags 1 and 2. In addition, there is a constant input term, which may be conveniently taken as 1 and this is connected to every neuron in the hidden and output layer. The introduction of this constant input unit, according to Adhikari and Agrawal (2013), avoids the necessity of separately introducing a bias term.

Fig. 1 The three-layer feed forward ANN architecture (Adhikari and Agrawal, 2013)

A new ANN model, which incorporates the issue of seasonal effect, was presented by Hamzacebi in 2008. The ANN model is known as the Seasonal Artificial Neural Network (SANN) model and is used for seasonal time series forecasting. The model is simple and also has been experimentally verified to be quite successful and efficient in forecasting seasonal time series.

3.3 Time Series Forecasting Using Support Vector Machines Based Models

The used of Vapnik’s support vector machine (SVM) based models for time series has been a major breakthrough in the area of modeling in time series forecasting. According to Adhikari and Agrawal (2013), Vapnik and his co – workers designed SVM at the AT & T Bell laboratories in 1995. The initial aim of SVM was to solve pattern classification problems but afterwards they have been widely applied in many other fields such as function estimation, regression, signal processing and time series prediction problems (Raicharoen et. al., 2003). The remarkable characteristic of SVM, according to Adhikari and Agrawal (2013), is that it is not only destined for good classification but also intended for a better generalization of the training data. This has made the SVM methodology to be one of the well-known techniques, especially for time series forecasting problems in recent years.

The objective of SVM is to use the structural risk minimization (SRM) principle to find a decision rule with good generalization capacity (Farooq, Guergachi and Krishnan, 2007). In SVM, the solution to a particular problem only depends upon a subset of the training data points, which are termed as the support vectors (Vapnik, 1998). Another important feature of SVM is that here the training is equivalent to solving a linearly constrained quadratic optimization. So the solution obtained by applying SVM method is always unique and globally optimal, unlike the other traditional stochastic or neural network models (Cao and Francis, 2003). Perhaps the most amazing property of SVM is that the quality and complexity of the solution can be independently controlled, irrespective of the dimension of the input space.

In SVM applications, the input points are usually mapped to a high dimensional feature space, with the help of some special functions, known as support vector kernels (Fan, Li and Song, 2006), which often yields good generalization even in high dimensions. The most popular of SVM based models for time series forecasting are: Least – square SVM (LS – SVM) and Dynamic Least – square SVM (DLS – SVM). The LS – SVM model employs the equality constraints and a sum-squared error (SSE) cost function for time series forecasting, instead of the quadratic program in traditional SVM (Adhikari and Agrawal, 2013). The DLS – SVM is the modified version of the LS – SVM model and is suitable for real time system recognition and time series forecasting. It employs the similar concept but different computation method than the recurrent LS – SVM (Suykens and Vandewalle, 2000). The key feature of DLS – SVM is that it can track the dynamics of the nonlinear time-varying systems by deleting one existing data point whenever a new observation is added, thus maintaining a constant window size.

4.0 New Learning in the Area of Time Series Forecasting Modeling Application

Time series forecasting models are statistical models used in time series analysis and forecasting. According to Zou and Yang (2003), one model is selected based on a selection of criterion, hypothesis testing, and/or graphical inspection. The selected model is then used to forecast future values. However, it is important to note that model selection is often unstable and may cause an unnecessarily high variability in the final estimation/prediction. Based on these facts, the new learning in the area of time series forecasting modeling application has been on combining the existing models for better performance of prediction.

Zhang in his 2003 work proposed a hybrid methodology that combines both ARIMA and ANN models in linear and non-linear modeling of time series forecasting. His experimental results with real data sets indicated that the combined model can be an effective way to improve forecasting accuracy achieved by either of the models used separately. Similar results were also obtained by Khashei and Bijari (2011) in their work titled “A Novel Hybridization of Artificial Neural Networks and ARIMA models for Time Series Forecasting”.

Time series forecasting using flexible neural tree model have also been achieved (Chen et. al., 2004). The proposed model, from architecture perspective was seen as a flexible multi-layer feed forward neural network with over-layer connections and free parameters in activation functions. The work demonstrated that FNT model with automatically selected input variables (time-lags) has better accuracy (low error) and good generalization ability. Simulation results for the time-series forecasting problems showed the feasibility and effectiveness of the proposed methods.

5.0 Conclusion

The three widely used model classes for time series forecasting have been identified as stochastic models, neural networks and the Vapnik’s support machine (SVM) based models. Hybridization of these models has been shown to improve or increase time series forecasting accuracy and generalization ability.

References

Adhikari, R. and Agrawal, R. K. (2013). An Introductory Study on Time Series Modeling and

Forecasting. Retrieved from: http://arxiv.org/ftp/arxiv/papers/1302/1302.6613.pdf

Campbell, M. J., Walker, A. M. (1977). A Survey of Statistical Work on the MacKenzie River

Series of Annual Canadian Lynx Trappings for the years 1821-1934, and a New

Analysis, J. R. Statist. Soc. Ser. A 140, 411-431

Cao, L. J. and Francis, E. H. (2003). Support Vector Machine with Adaptive Parameters in

Financial Time Series Forecasting, IEEE Trans. on Nueral Networks, vol. 14, No. 6,

1506-1518

Chartfield, C. (1996). Model Uncertainty and Forecast Accuracy, Journal of Forecasting, 15,

495-508

Chen, Y., Yang, B., Dong, J. and Abraham, A. (2004). Time Series Forecasting Using Flexible

Neural Tree Model, Information Sciences, 1-17

Fan, Y., Li, P. and Song, Z. (2006). Dynamic Least Square Support Vector Machine,

Proceedings of the 6th World Congress on Intelligent Control and Automation (WCICA),

Dalian, China, 4886-4889

Farooq, T., Guergachi, A. and Krishnan, S. (2007). Chaotic Time Series Prediction Using

Knowlege Based Green’s Kernel and Least-squares Support Vector Machines, Systems,

Man and Cybernetics, ISIC, 7-10, 373-378

Hipel, K. W. and McLeod, A. I. (1994). Time Series Modelling of Water Resources and

Environmental Systems, Amsterdam, ElservierHamzacebi, C. (2008). Improving Artificial Neural Networks’ Performance in Seasonal Time

Series Forecasting, Information Science, 178, 4550-4559

Khashei, M. and Bijari, M. (2011). A Novel Hybridization of Artificial Neural Networks and

ARIMA Models for Time Series Forecasting, J. of Appl. Soft Computing, 11, 2664-2675

Parrelli, R. (2001). Introduction to ARCH & GARCH Models, Optional TA Handouts, Econ

472, Department of Economics, University of Illinois

Raicharoen, T., Lursinap, C. and Sanguanbhoki, P. (2003). Application of Critical Support

Vector Machine to Time Series Prediction, Circuits and Systems, ISCAS 03, Proceedings

of the 2003 International Symposium, Vol. 5, 25-28

Suykens, J. A. K. and Vandewalle, J. (2000). Recurrent Least Squares Support Vector Mechines,

IEEE Trans. Circuits System-I, Vol. 47, no. 7, 1109-1114

Vapnik, V. (1998), Statistical Learning Theory, New York: Wiley

Zhang, G., Patuwo, B. E. and Hu, M. Y. (1998). Forecasting with Artificial Neural Networks:

The State of the Art, International Journal of Forecasting, 14, 35-62

Zhang, G. P. (2003). Time Series Forecasting using a Hybrid ARIMA and Neural Network

Model. Journal of Neurocomputing, 50, 159-175

Zhang, G. P. (2007). A Neural Network Ensemble Method with Jittered Training Data for Time

Series Forecasting, Information Sciences, 177, 5329-5346

Zou, H. and Yang, Y. (2004). Combining Time Series Models for Forecasting, Int. J. of

Forecasting, 20, 69-84