Muhannad A. El-Mefleh, National University

This paper compared the forecast accuracy of GDP by using an econometric model, the auto-regressive integrated moving-average (ARIMA) model, the average method, the minimum variance approach, the odd-matrix (weighted average) method, and the simple linear composite method. The Mean Absolute Percentage Forecast Inaccuracy (MAPFI) method has been used to measure the precision of the models’ forecasts. The major findings of the paper are that (1) each one of the econometric model, the minimum variance approach, and the odd-matrix approach was superior to all other methods in four out of 15 forecasted periods; (2) only once out of 15 forecasted periods did the average forecast model outperform the other methods; (3) the simple linear combined method outperformed the other methods two times out of 15 forecasted periods; (4) the ARIMA model was inferior to all other methods; (5) the minimum variance approach produced the smallest MAPFI comparative to all other methods; and (6) the minimum variance approach performed slightly better than any of the other methods.

This paper is devoted to finding a forecasting combination method that is superior to the econometric model forecast or the forecast of ARIMA model when applied to Jordan's GDP data. There are many ways of combining different forecasting methods. Four major composite methods will be considered in  this paper, but one method will be identified as the superior one. These methods are the equal weight,  the minimum variance, the odds-matrix, and the regression approach.

McNees (1985) argued that the experience of the past  tells us that no one forecasting model remains accurate for all variables all the time, and that the most accurate  forecast in the short run is not necessarily the most accurate in the long run.  The statistical tests for a  model's performance are conditional because they are based on the assumption that the model is correctly specified. It is true that the model should have an acceptable theoretical basis; unfortunately, acceptable theoretical bases differ between economists.  Therefore, the selection of a forecasting model cannot be accomplished by simply choosing  the best model for a given period. Since identifying the best model for each period in most cases is not feasible, combining different forecasts, which average differences when one measure gives an over-forecast while the other an under-forecast, seems a sensible alternative. Errors may cancel each other out so that the combined forecast would turn out to be relatively closer to the actual value than either forecast independently.

Winkler and Makridakis (1983) and Clemen (1989) found that the accuracy of combined forecasts rose with the increase of the number of forecasting methods. However, combining more than five methods of forecasting to the combined model will not improve the accuracy of the forecast. A weighted method allows the consideration of relative accuracy of one method over the other; therefore, it can improve on the average combining method. Winkler and Makridakis (1983) showed that combining forecast is superior to all individual methods in the short run and to most of the methods in the long run.

Also, when uncertainty exists about which model is better, then combined forecasts improve the forecast.  When one model is superior in performance to the other for each  period, then combining forecasts should not be considered because we choose directly the best model. Nevertheless, the risk of not choosing the best single method for forecasting can be very serious. This risk decreases rapidly when combined forecasts are considered. Thus, instead of choosing one forecasting method, it is reasonable to consider a forecast combination that contains more information than any given single method.  Granger (1980) encouraged the combined forecast, especially when different forecasts are based on different philosophies, as is the case of the regression and the Box-Jenkins forecast.

Bates and Granger (1969) argued that discarding the inferior forecasting methods will lead to the discarding of useful information that the superior forecast method may not consider. He demonstrated that the combined forecasts could yield an improvement over the individual forecast. Diebold (1989) argued that when the model builder is the forecaster, the possibility of combining information rather than forecast will be improved. Diebold (1990) argued that combining information would be superior to combining forecasts, but combining information may not be feasible or cheap.

The first procedure is the equal weight method, which is the  easiest one to calculate and is accomplished by taking the average of both forecasts.  This method does not require any knowledge about the accuracy or the correlation between  the two errors.  This procedure, according to Clemen and Winkler (1986), serves as a benchmark and performs better than some other more complicated theoretical methods such as the minimum variance approach and the regression approach.  Clemen and Winkler (1986) used the simple average of GNP forecasts by utilizing the four major models of Wharton econometrics, the Chase Econometrics, the Data Resources, Inc., and the Bureau of Economic Analysis.  Clemen and Winkler (1986) found that the simple average performed better than any single model, and better than the normal model combining methods which treat the vector of forecast error as being normally distributed with no autocorrelation in the error terms.   The criterion used for the performance accuracy was the  mean square error and the mean absolute deviation.

Clemen (1989) in his review of the combined forecasting methods observed that combining forecasting is widely used not only in economics but also in management science and psychology. He cited psychological studies where the average of judgements is superior to individual judgement in a clinical situation. Clemen (1986) showed that simple average forecast of GNP performed better than other sophisticated forecasting methods.

Makridakis and Winkler (1983) explore the performance of a simple average forecast for ten different methods.  They find that the equal weight outperformed all of the sixteen individual methods they considered. Where the mean absolute percentage error (MAPE) had been calculated for all  kinds of forecasts, they demonstrate that MAPE, the variance of MAPE, and the average of MAPE decline as the number of methods included in the combined forecast increases. Also the average of MAPE is smaller for large numbers  of combination forecast methods than for small  numbers, in the long run.

Finally, the simple average forecast error will be equal  to the  average  error of the combined forecast. Thus,  if  the  two forecast methods that will be used for the Jordanian data have no consistent biases, the error series can be considered white noise, and the combined forecast is unbiased.

The second method is the minimum variance approach.   Granger (1980) explained this method in the following manner: when both forecasts for Jordanian data have no consistent bias, then

Combine forecast (c) = k (forecast of Box-Jenkins) + (l-k) (regression forecast)

Combine error (c(e)) = k (forecast error of Box-Jenkins) + (l-k) (forecast error of regression)

Var (c(e)) = k² var (forecast error of Box-Jenkins) + (l-k)² (var (forecast error of regression)) + 2k (l-k) (cov (e_Box-Jenkins, e_regression))

Getting the value of k that minimizes the var (c(e)), requires taking the partial derivative of var (c(e)) with respect to k leads to

K = (Var (e_regression) - Cov (e_Box-Jenkins, e_regression)) / ( Var (e_Box-Jenkins) + Var (e_regression) -2 Cov (e_Box-Jenkins, e_regression))

This  method  of calculating k will give us the variance  of  the combined  forecast  error, which is smaller than any one of  the original forecasts.

The  third  method is the odds-matrix (OM) method, which  is suggested  by Gupta and Wilton (1987).  This method is based on the outperformance measure which was suggested by Bunn (1975), where model one is considered to outperform model two  if  model one has a smaller absolute error.  We then give a weight for each model equal to the percentage of times that model outperforms the other  through the available data set.  Let Z₁ and  Z₂  represent the  percentage of times that model one and model two  outperform each  other,  respectively, during the available data  set.  Then calculate the following:

W₁= z₁/(z₁ + z₂)

W₂= z₂/(z₁ + z₂)

Where W₁ represents the odds that model one will outperform model two, and W₂ represents the odds that model two will outperform model one. After getting the value of w^*, we apply this weight to forecast beyond the data set.  This combined approach is expandable to n numbers of forecast methods. Also,  this  odds-matrix approach is expected to perform reasonably  well because its  weight is insensitive to small changes in odd ratios.  Thus, one does not need large amounts of prior performance data to perform well.  This method also permits  updating which reduces the influence of a given model whose forecast deteriorates over time. Finally, this method  offers a simple approach for combining forecasts.

From  examples given by Gupta and Wilton (1987) for these three different hypothetical models, the combination forecast produces lower forecast error than the  single  best model by using the mean average error criteria.  The relative overall accuracy of the odds-matrix approach did  exceed the equal weight, the minimum variance approach, and regression for large size samples and did surpass the other methods when the sample size was limited (such as five or  ten). The reduction in error also stabilizes faster when we use the OM approach rather than the other methods as data  increases.  The only exception to the above result is that when the model variances are equal, the equal weight method produces  smaller errors than the minimum variance or the OM approaches.

In another article, Gupta and Wilton (1988) used the OM method of combining forecasts and applied it to Clemen and Winkler's (1986) data. The OM approach proved to be more accurate than all the methods examined by Clemen and Winkler  (1986), including the simple average forecast which is based on equal weight.  Gupta and Wilton (1988) calculated the forecast error over each time series and then calculated the mean square error (MSE) and the mean average deviation (MAD)  for the OM method, demonstrating a substantial reduction in the MSE and MAD compared with the other methods.  This method provides a considerable improvement in forecast accuracy and makes a meaningful combination possible. This combination is not always possible for other methods like the normal model because the weight may be a negative value or larger than one for some models.  The OM  method is stationary when the average MAD or MSE for a relatively large data set is relatively comparable to a small data set.  If the above is not the case, however, then it implies that the error is a nonstationary process.  When one has a nonstationary forecast error process, then the  OM  method  can perform better than that of the equal weight forecast by using a small number of recent observations.

The fourth method of combining forecasts is the linear composite prediction, which is explained by Nelson (1973) in the following manner:

A_t = B₁ (Box-Jenkins forecast_t) + B₂ (econometric model forecast_t) + e_t

Where: A_t actual value for period t and e_t is composite forecast error.

An OLS estimate will then provide the forecaster with the  minimum  mean square error linear combined forecast for the sample period. When each model forecast is unbiased, then B₂ =1 - B₁, and the model can be rewritten in the following manner:

A_t  - econometric model forecast = B₁ (Box-Jenkins forecast_t - econometric forecast_t) +e_t

with OLS estimating B₁.

If the Box-Jenkins forecast subsumes the econometric model forecast, and one has a large sample,  the Plim  B₁  =  0.  If the econometric model forecasts do incorporate  the information provided by Box-Jenkins, however, then with large samples B₁ will approach unity. In practice, the forecaster can add a constant to the above model and compare the forecast accuracy of each one and then choose the best.

Deutsch, Granger & Terasvirta (1994) argued that the least square regression is easy and produces a combined forecast superior to either of the individual forecasts. Nonlinear combined procedures resulted in better forecast performance over the simple linear combining approach.

Diebold (1988) showed that linear regression combined forecast may lead to serially correlated error. Therefore, using the combined forecast equation with the auto regressive of order p [AR (P)] for the disturbance term will improve the forecast accuracy.

Coulson (1993) found that the inclusion of lagged dependent variables, not lagged forecasts, improved forecast accuracy.

Y_t=b₀+b₁ f₁ +b₂ f₂ +b₃ y_t-1 + e_t

is superior to

Y_t=b₀+b₁ f₁ +b₂ f₂ + e_t

Where f₁ is forecast of model one

f₂ is forecast of model two.

Therefore, the inclusion of a source of dynamic process through the use of lagged dependent variables in forecast combination will improve the post-sample forecast.

The econometric model of Jordan that was constructed by El-Mefleh (1999) and the ARIMA (1,2,0) model were used for the ex-post forecast for one year after the data had been used for the 1983-1997 period. The following combined forecast methods were used:

1. Equal weight method

2. The minimum variance approach

   Combine forecast (c) = k (forecast of Box-Jenkins) + (l-k) (regression forecast)

   Combine forecast (c) = .2596 (forecast of Box-Jenkins) + .7404 (regression forecast)

   Where

   K= (Var (e_regression) - Cov (e_Box-Jenkins, e_regression)) / (Var (e_Box-Jenkins) + Var (e_regression) -2 Cov (e_Box-Jenkins, e_regression))

   Var (e_regression) = 4812.543

   Var (e_Box-Jenkins) = 28109.569

   Cov (e_Box-Jenkins, e_regression) = 7190.551

3. The odd-matrix approach

Let Z₁ and  Z₂  represent the  percentage of times that model one and model two  outperform each  other,  respectively, during the available data  set.  Then calculate:

W₁= z₁/(z₁ + z₂) = .2

W₂= z₂/(z₁ + z₂) = .8

Where W₁ represents the odds that the ARIMA model outperformed the econometric model, and W₂ represents the odds that the econometric model outperformed the ARIMA model.

4. The linear composite prediction

The fourth method of combining forecasts, the linear composite prediction, is explained by Nelson (1973) in the following manner:

A_t = B₀ + B₁ (Box-Jenkins forecast_t) + B₂ (econometric model forecast_t) + e_t

A_t = 95.126 + 1.119(Box-Jenkins forecast_t) - .156 (econometric model forecast_t)

t-value 2.236 10.135 -1.406

The Mean Absolute Percentage Forecast Inaccuracy are shown in Table 1 and the forecasted values of GDP by six approaches are shown in Table 2.

Minimum variance	.008936
Econometric model	.01122
Linear combination	.012113
Odd-matrix	.01354
Equal weight	.02256
ARIMA model	.03926

The major findings of this paper are that the forecast of the econometric model was superior to all other methods in four out of 15 forecasted periods. The minimum variance approach was superior to all other approaches in four out of 15 forecasted periods. The odd-matrix approach performed better than the other methods in four out of 15 forecasted period. The simple linear combined method outperformed the other approaches in two times out of 15 forecasted periods. Finally, the average forecast was superior only one time to all other methods.

Bates J. M. and C. W. J. Granger (1969) "The Combination of Forecasts" Operational Research Quarterly, Vol. 20 (4), 451-468.

Clemen, R. T. & Robert L. Winkler (1986) "Combining Economic Forecasts" Journal of Business & Economic Statistics, 4 (1), 39-46.

Clemen, R. T. (1989) "Combining Forecast" International Journal of Forecasting 5. 559-583.

Coulson, N. E. (1993) "Forecast Combination in a Dynamic Setting" Journal of Forecasting, 12, 63-672.

Deutsch, Melinda, Clive W.J. Granger, Timo Terasvirta (1994) "The Combination of Forecasts Using Changing Weights" International Journal of Forecasting 10, 47-57.

Diebold, F. X. (1988) "Serial Correlation and the Combination of Forecasts" Journal of Business & Economic Statistics", 6 (1), 105-111.

Diebold, F. X. (1989) "Forecast Combination and Encompassing: Reconciling Two Divergent Literatures" International Journal of Forecasting 5, 589-592.

Diebold, F. X. (1990) "The Use of Prior Information in Forecast Combination" International Journal of Forecasting 6, 503-508.

El-Mefleh, M. (1999) "Econometric Forecasting Model For Jordan" Journal of the American Association of Behavioral & Social Sciences, 3, 98-119.

Granger, C. W. J. (1980) "Forecasting in Business and Economics" (New York: Academic Press).

Gupta, S., and Peter C. Wilton (1987) "Combination of Forecasts: An Extension", Journal of Management Science, 33 (3), 356-372.

Gupta, S., and Peter C. Wilton (1988) "Combination  of Economic Forecasts: An Odds-Matrix Approach", Journal of Business and Economic Statistics, 6 (3), 373-379.

Makridakis,  S., and Robert L. Winkler (1983) "Average of Forecast: Some Empirical  Results", Journal of  Management Science, 29 (9), 987-996.

McNees S. K. (1985) "Which Forecast Should you Use", New England Economic Review (Boston: Federal Reserve Bank of Boston), July/August, 36-42.

Nelson, C. R. (1973) Applied Time Series Analysis for Managerial Forecasting. San Francisco: Holden-day.

Winkler, Robert and Spyros Makridakis (1983) "The Combination of Forecasts" Journal of the Royal Statistical Society. Series A, 150-157.