One of the most challenging parts of my project for GSoC is to create the functionality of time series cross-validation. Model Evaluation metrics are highly critical to check how your model performs on real life and unseen data. The model fit metrics(like AIC, BIC) are quite enough to know how your model forecasts on new data as these metric only use the model’s fit data.
Cross-validation is one of the widely accepted evaluation metrics to judge how a model responds to new data.
Cross-validation is a technique to evaluate predictive models by partitioning the original sample into a training set to train the model, and a test set to evaluate it.
For classification problems, one typically uses stratified k-fold cross-validation, in which the folds are selected so that each fold contains roughly the same proportions of class labels.
In repeated cross-validation, the cross-validation procedure is repeated n times, yielding n random partitions of the original sample. The n results are again averaged (or otherwise combined) to produce a single estimation.
In Time Series Cross Validation, there is a series of test sets, each consisting of a single observation. The corresponding training set consists only of observations that occurred prior to the observation that forms the test set. Thus, no future observations can be used in constructing the forecast.
For time series forecasting, a cross-validation statistic is obtained as follows
- Fit the model to the data y1,…,yt and let ^yt+1 denote the forecast of the next observation. Then compute the error (e∗t+1=yt+1−^yt+1) for the forecast observation.
- Repeat step 1 for t=m,…,n−1 where m is the minimum number of observations needed for fitting the model.
- Compute the MSE from e∗m+1,…,e∗n.