BUS606: Operations and Supply Chain Management

Forecasting accuracy is measured using the mean absolute percentage error depicted in (11). Despite its criticism, the MAPE remains a popular and recommended error measure. In this study, the applicability of the measure is not compromised by scale incompatibilities, negative or close to zero values. Hence, there is no compelling reason to apply alternative measures that aim at addressing potential shortcomings for these specific cases. It is however acknowledged that MAPE puts a heavier penalty on positive forecasting errors than on negative forecasting errors. The sometime suggested Symmetric Mean Percentage Error (SMAPE) has instead a heavier penalty on negative forecasting errors and therefore provides no considerable alternative:

$\text { MAPE }=\left(\frac{1}{T} \sum_{t=1}^{T}\left|\frac{Y_{1}-F_{1}}{Y_{1}}\right|\right) \times 100 \%$

Forecasting accuracy of the ARIMA model and the vector time series model are compared as well as contrasted with two simple models that were included as benchmark. One is the naive model, which is often considered for this type of comparison. Like a random walk model, it uses the most recent observation as a predictor. The other model, a seasonal naive model, uses the last observation of the same season as predictor.

The accuracy of each model is measured and compared for the in-sample $\mathrm{t}=\{1, . ., 731\}$ and the holdout sample $\mathrm{t}=\{732, . ., 759\}$. In order to obtain more robust results, the holdout period of 28 days is split further in two periods of 14 days. In addition, the onestep ahead non-cumulative error is compared with the cumulative error for the holdout samples $\mathrm{t}=\{732, . ., 745\}$ and $\mathrm{t}=\{746, . ., 759\}$.

Results of the various models and sample periods are summarized in Table 2. Clearly, the vector time series models perform best for the in-sample period and outperform the SARIMA models.

Table 2: Forecasting errors (MAPE)

Notes: Results depict the mean absolute percentage error for the network of 20 ATMs and compare the 12 models for time periods $\mathrm{t}=\{1, . ., 731\}$ (March 21st, 2007-March 20th, 2009), $\mathrm{t}=\{732, . ., 745\}$ (March 21st, 2009-April 3rd, 2009) and $\mathrm{t}=\{746, . ., 759\}$ (April 4th, 2009-April 17th, 2009)

Exogenous variables that capture calendar effects further reduce the errors of SARIMA models and vector time series models. Limiting the calendar effects under consideration to the day-of-the-week and month-of-the-year or simply the day-of-the-week results in larger MAPEs. Nevertheless, these models outperforms their counterparts without exogenous variables. The VAR(7) model yields a higher accuracy during the in-sample period than the VAR(1) model. This effect diminishes and reverses for the models including exogenous variables during the holdout period.

However, the vector time series model VAR(7) without exogenous variables results in lower MAPEs than VAR(1) suggesting that an over-specified model can reduce forecasting errors.

The result for the non-cumulative holdout sample is not as unambiguous as for the in-sample period. MAPEs for the holdout period are generally lower, but increase during the second half. This indicates that holiday effects present in $\mathrm{t}=\{746, . ., 759\}$ are only partially accounted for and may in fact be determined by additional factors such as weather and festivals. In fact, the holiday effect is reversed during the holdout period with more cash being dispensed than on an average day or compared to the in-sample period.

The seasonal naive model outperforms the nonseasonal naive model during the in-sample as well as the holdout period. MAPEs' of the seasonal naive model are in line with the SARIMA model and vector time series models suggesting that a large part of the variability in the demand series is attributed to seasonality in form of the day-of-the-week effect.

Moreover, cumulative forecasts result in exceptionally low MAPEs suggesting that most forecasting errors are offset over a period of 14 days. Cumulative 14 days forecasts using the naive method match the performance of the SARIMA model without exogenous variables despite the non-cumulative errors being more than twice as large. Joint forecasting and exogenous variables do not further reduce MAPEs for cumulative forecasts.