Measures of Forecast Error

As mentioned earlier, every instance of demand has a random component. A good forecasting method should capture the systematic component of demand but not the random component. The random component manifests itself in the form of a forecast error. Forecast errors contain valuable information and must be analyzed carefully for two reasons  

Managers use error analysis to determine whether the current forecasting method is predicting the systematic component of demand accurately. For example, if a forecasting method consistently produces a positive error, the forecasting method is overestimating the systematic component and should be corrected. 

All contingency plans must account for forecast error. Consider a mail-order company with two suppliers. The first is in the Far East and has a lead time of two months. The second is 

As defined earlier, forecast error for Period t is given by E(, where the following holds  

That is, the error in Period t is the difference between the forecast for Period t and the actual demand in Period t. It is important that a manager estimate the error of a forecast made at least as far in advance as the lead time required for the manager to take whatever action the forecast is to be used for. For example, if a forecast will be used to determine an order size and the supplier s lead time is six months, a manager should estimate the error for a forecast made six months before demand arises. In a situation with a six-month lead time, there is no point in estimating errors for a forecast made one month in advance. 

---

A common measure of forecast error is the mean absolute deviation (MAD). The MAD is easily calculated and is easily interpreted. The MAD is calculated by adding the absolute value of the forecast errors each period (I Demand - Forecast ) and then taking the average of this total. This is illustrated in Table 8.1. 

Another measure of forecast error is mean squared error (MSE), which is the average of the square of total forecast errors for a sample. This approximates the variance. The formula to calculate the MSE is ... 

In fact, Z/n is called the mean absolute deviation (MAD) in forecasting and is used as a key measure of forecast error in validating the forecasting method (refer to Section 2.9). For the LP model, the unrestricted variable g( will be replaced by the difference of two non-negative variables as follows ... 

Let us illustrate all the five measures of forecast errors with an example. 

It is recommended that multiple measures of forecast errors be used in selecting the best forecasting method. If a particular method consistently does well in all the measures, then it is clearly the best method to use. However, it is possible that some methods may do well in some measures and pKJorly in other measures. In such situations, it is quite common to choose the best two or three methods and use the average of their forecasts as the forecast for the future. 

Forecasts are always inaccurate and should thus include both the expected value of the forecast and a measure of forecast error. To understand the importance of forecast error, consider two car dealers. One of them expects sales to range between 100 and 1,900 units, whereas the other expects sales to range between 900 and 1,100 units. Even though both dealers anticipate average sales of 1,000, the sourcing policies for each dealer should be very different, given the difference in forecast accuracy. Thus, the forecast error (or demand uncertainty) is a key input into most supply chain decisions. Unfortunately, most firms do not maintain any estimates of forecast error. 

One measure of forecast error is the mean squared error (MSE), where the following holds (the denominator in Equation 7.21 can also have n -I instead of n) ... 

The MAPE is a good measure of forecast error when the underlying forecast has significant seasonality and demand varies considerably from one period to the next Consider a scenario in which two methods are used to make quarterly forecasts for a product with seasonal demand that peaks in the third quarter. Method 1 returns forecast errors of 190, 200, 245, and 180 Method 2 returns forecast errors of 100,120,500, and 100 over four quarters. Method 1 has a lower MSE and MAD relative to Method 2 and would be preferred if either alterion was used. If demand is highly seasonal, however, and averages 1,000, 1,200, 4,800, and 1,100 in the four periods. Method 2 results in a MAPE of 9.9 percent, whereas Method 1 resnlts in a mnch higher MAPE, 14.3 percent. In this instance, it can be argued that Method 2 should be preferred to Method 1. 

A forecast error is equal to the difference between genuine or real and predicted or forecast values. As in other cases, in its value may be identified two addends correctness, i.e., closeness of the predicted value to the genuine one, and precision as a measure of random errors with equal probability of variance from some medium value. The former ones may be eliminated, the latter ones may only be decreased. 

Let us discuss each of the measures in detail, assuming we have n values of forecast errors, Cy Cy... 

Mean absolute percentage error (MAPE) MAPE measures the relative dispersion of forecast errors and is given by... 

Semi-automatic software These are moderately priced but they require the user to have some basic knowledge of forecasting principles and fechniques. Here, fhe user has to select an appropriate forecasting technique based on the analysis of time series data. The software will then compute the optimal parameters for the chosen method using some measure of forecasf error. It also gives the forecasts and all the statistics, such as MAD, MAPE, MSE, Bias, etc. The software makes no recommendation as to which forecasting technique is appropriate for the given data. 

As discussed in Chapter 7, demand has a systematic as well as a random component. The random component is a measure of demand uncertainty. The goal of forecasting is to predict the systematic component and estimate the random component. The random component is usually estimated as the standard deviation of forecast error. We illustrate our ideas using uncertain demand for a smartphone at B M Office Supplies as the context. We assume that periodic demand for the phone at B M is normally distributed with the following inputs ... 

The optimal order size is obtained using Equation 13.2 and the expected profit using Equation 13.3. The order size and expected profit as forecast accuracy (measured by standard deviation of forecast error) varies are shown in Table 13-3. 

To choose k test stores, we partition the n stores of the chain into k clusters. The stores within each cluster are chosen to minimize a measure of dissimilarity based on the percent of total sales represented by sales of each of the prior products in each store. Two stores that sold exactly the same percentage of each of the prior products would be in the same cluster, and all of the stores within a cluster would sell approximately the same percentage of each of the prior products. We then choose a single test store within each cluster that best represents the cluster in the sense that using test sales at this store to forecast sales of other stores in the cluster minimizes the cost of forecast errors. 

MSD stands for Mean Squared Deviation. MSD is always computed using the same denominator, n, regardless of the model, so we can compare MSD values across models. MSD is a more sensitive measure of an unusually large forecast error than MAD. 

No accuracy measure is generally applicable to all forecasting problems due to variation in forecasting objectives and data scales (De Gooijer and Hyndman, 2006 Hyndman and Koehler, 2006). Let denote the observation at time t and F, denote the forecast of Y. Then define the forecast error e=Y F. In this chapter,... 

According to Armstrong and Fildes (1995), the objective of a forecast accuracy measure is to provide an informative and clear understanding of the error distribution. Theoretically, when the forecast errors are randomly structured, the form of the forecasts is independent of the selected accuracy measure. Otherwise, it is generally accepted that there is no single best accuracy measure, and deciding on the assessment method is essentially subjective. In this study, a simple form of relative error (E) is selected as the forecast accuracy measure, since it offers a number of desirable properties ... 

The empirical assessment of experts relative error of estimates revealed that over 45% of errors were close to one (expert estimate true value). Additionally, lognormal was identified as one of the best fitted distributions, considering the selection of relative error as the forecast accuracy measure. The study also showed 285% average improvements in experts estimates with 77% of estimates improved, applying the likelihood function developed by relative errors for homogenous and nonhomogenous cases. 

Ahlburg, D. A. 1992. Error measures and the choice of a forecast method. IntemationalJoumal of Forecasting, 8 99-111. D. 

Armstrong, S. R. Fildes 1995. On the selection of error measures for comparisons among forecasting methods. Journal of Forecasting, 14 67-71. 

For clarity, this example first calculated the forecast error. It then took the absolute value of that forecast error, and, finally, the average of the absolute value was calculated. The MAD gives us a measure of the distribution of the forecast error, so we can estimate a minimum estimate and a maximum estimate for demand. For example, if our forecast model forecast 130 for period 6, we would know that that if the actual demand was within 1 MAD from the point estimate of demand (i.e., 130) it could be as high as 140 and as low as 120. Statisticians have calculated that 1 MAD = 0.80 standard deviations, so that 3.75 MADs is equivalent to 3.00 standard deviations (see Melnyk and Denzler, 1996, for more information). Because 3.0 standard deviations includes 99.87% of the population, when forecasters provide a forecast and the value of one MAD, they allow the user to quickly determine the accuracy of the forecast. This is illustrated in Figure 3.2. In this example, 3.0 standard deviations means that there is a 99.87% chance that the actual demand will be between 39 of the forecast of 130. Or, the actual demand has a 99.87% chance of being between 91 and 169. This is calculated as 130 (3.75) (10.4). 

It is important to distinguish systematic error from random error. To do this, the forecaster measures the amount of bias in the forecast. The APICS Dictionary defines bias as A consistent deviation from the mean in one direction (high or low). A normal property of a good forecast is that it is not biased. The usual measure of bias is the mean forecast error (MFE) which is the mean of the difference between the actual demand and the forecast demand. The formula for the MFE is ... 

The accuracy of the forecast depends on forecast errors, which measure the differences between the forecasted demands and their actual (observed) values. As discussed in Section 2.4.7, the/orecast error, for period t, denoted by e is given by ... 

Mean absolute deviation (MAD) MAD measures the dispersion of the forecast errors ... 

There needs to be a structured agenda and agreed set of measures used - most typically including forecast error, forecast volumes and sales quantities, capacity utilisation, new products and upcoming events, and sku review. 

Forecast error measures the difference between the forecast and actual demand. The forecast error is a measure of uncertainty and drives all responses to uncertainty, such as safety inventory or excess capacity. 

Companies should estabhsh clear performance measures to evaluate the accuracy and timeliness of the forecast. These measures should be linked to the objectives of the business decisions based on these forecasts. Consider a mail-order company that uses a forecast to place orders with its suppliers, which take two months to send in the orders. The mail-order company must ensure that the forecast is created at least two months before the start of the sales season because of the two-month lead time for replenishment. At the end of the sales season, the company must compare actual demand to forecasted demand to estimate the accuracy of the forecast. Then plans for decreasing future forecast errors or responding to the observed forecast errors can be put into place. 

We then estimate that the mean of the random component is 0, and the standard deviation of the random component of demand is a. MAD is a better measure of error than MSE if the forecast error does not have a symmetric distribution. Even when the error distribution is symmetric, MAD is an appropriate choice when selecting forecasting methods if the cost of a forecast error is proportional to the size of the error. 

Keep in mind that none of these tools is foolproof. Forecasts are virtually always inaccurate. A good IT system should help track historical forecast errors so they can be incorporated into future decisions. A well-structured forecast, along with a measure of error, can significantly improve decision making. Even with aU these sophisticated tools, sometimes it is better to rely on human intuition in forecasting. One of the pitfalls of these IT tools is relying on them too much, which eliminates the human element in forecasting. Use the forecasts and the value they deliver, but remember that they cannot assess some of the more qualitative aspects about future demand that you may be able to do on your own. 

Analyze demand forecasts to estimate forecast error. Forecast error measures the random component of demand. This measure is important because it reveals how inaccurate a forecast is likely to be and what contingencies a firm may have to plan for. The MSE, MAD, and MAPE are used to estimate the size of the forecast error. The bias and TS are used to estimate if the forecast consistently over- or underforecasts or if demand has deviated significantly from historical norms. 

---