Forecast Evaluation Exercises in R: 50 Practice Problems

Explanation: Forecast error is conventionally actual minus forecast, so a positive value means you under-forecast and a negative value means you over-forecast. Keeping that sign convention matters: reverse it and every bias diagnostic later in this hub flips direction. R subtracts the two columns element by element, so no loop is needed.

Explanation: Mean error (ME) measures bias, not accuracy. A value near zero says over- and under-forecasts roughly balance out, which is what you want, but a forecast that misses by +50 and -50 alternately also scores zero. That is why ME is always read next to MAE or RMSE, never alone.

Explanation: MAE reports the typical miss in the units of the series, here five units of demand per week. Compare it with the ME of 0.75 from the previous exercise: the forecast is nearly unbiased yet still off by five units a week, which is the standard reason to report both numbers together. Naming the value with c(MAE = ...) keeps the output self-labelling.

Explanation: RMSE is always greater than or equal to MAE, and the gap widens as the errors become more uneven. Here RMSE is 5.41 against an MAE of 5, a small gap because no single week blows out. Optimising RMSE targets the conditional mean of the series, while optimising MAE targets the conditional median.

Explanation: Dividing by the actual makes errors comparable across weeks of different volume, which is why percentage errors travel well between products or regions. The catch appears whenever an actual is near zero: the denominator collapses and the percentage explodes. Exercise 3.2 puts a number on how badly that distorts the average.

Explanation: Naming the elements inside c() gives you a self-documenting result you can index by name later, as in ex_1_6[["RMSE"]]. Computing e once avoids recomputing the subtraction three times, and the same skeleton extends cleanly to MAPE or MASE. The forecast package ships accuracy() for exactly this job, covered in Section 4.

Explanation: Identical MAE, RMSE nearly triple. Squaring gives the lone 24-unit miss enormous weight, so RMSE is the metric to choose when one catastrophic error costs far more than several small ones, such as a stockout. When all misses cost the same per unit, MAE is the honest summary and RMSE will mislead you.

Explanation: The $mean element of a forecast object holds the point forecasts as a ts aligned to the test period, so subtracting nile_test matches observations to forecasts by date automatically. An MAE of 128 is meaningless on its own; it only becomes interpretable next to the series level of roughly 900, or next to a rival method.

Explanation: The mean method wins here, which is a useful surprise: WWWusage rises then falls back, so the long-run average lands closer than the last observed value. Never assume naive is the strongest baseline. Running both costs one extra line and occasionally saves you from shipping a model that loses to an average.

Explanation: Drift is slightly worse, so the extra parameter bought nothing. The Nile record has a level shift around 1900 rather than a steady trend, and drift fitted over the whole history extrapolates a slope that is not really there. Added flexibility must earn its keep on held-out data, not on the training fit.

Explanation: RMSE exceeds MAE by about 8 percent, a modest gap saying no single month blows out. Seasonal naive is the benchmark every serious model on this series must beat, because it captures the annual cycle for free. Note that it ignores the upward trend entirely, which is why the errors grow with horizon in Exercise 2.6.

Explanation: The WWWusage MAE is smaller in absolute terms, yet relative to a series that sits around 200 it is a far worse miss than 128 on a series that sits around 900. Scale-dependent metrics compare models on one series only. Section 3 introduces MAPE and MASE, which are built to cross that boundary.

Explanation: as.numeric() strips the time-series attributes so the result is a plain column rather than a ts that data.frame() would mangle. A single average MAE hides this profile entirely: a model that is excellent one month out and hopeless at twelve can post the same headline number as a mediocre but stable one.

Explanation: Every one of the worst five horizons sits in the second forecast year, which is the fingerprint of a method that ignores trend: seasonal naive repeats 1958 twice while traffic keeps climbing. Diagnosing failure by horizon tells you what to fix, whereas a single MAE only tells you that something is wrong.

Explanation: The forecast is off by about 15.5 percent on average, a statement any stakeholder understands without knowing the series scale. Take the absolute value of the ratio, not of the numerator alone, or negative actuals would silently flip signs. Exercise 4.2 confirms this figure against the MAPE column of accuracy().

Explanation: Four of the five rows are forecast to within 10 percent, yet the reported MAPE is 73.3 because one row with an actual of 0.5 contributes 300. This is why MAPE is dangerous for intermittent demand, new products or any series that can dip near zero. Use MASE or a scaled error there instead.

Explanation: sMAPE halves the damage, 35.2 against 73.3, because the forecast of 2 keeps the first denominator away from zero. The 200 in the numerator is not a typo: dividing by the sum rather than the mean of the two values needs a factor of two to stay on a percentage scale. sMAPE is still unstable when both values approach zero, and it is not truly symmetric, so treat it as a patch rather than a cure.

Explanation: A MASE just under 1 means the ten-year-ahead naive forecast is marginally better than a one-step naive forecast was in training, which is close to a coin flip. The diff() call gives period-to-period changes, and averaging their absolute values gives the natural difficulty scale of the series. MASE has no zero-denominator problem unless the series never changes.

Explanation: A MASE of 2.49 says the 24-month-ahead forecast is roughly two and a half times worse than an in-sample one-step seasonal naive forecast, which is expected given the long horizon. Getting the lag wrong is the classic MASE bug: using lag = 1 on seasonal data produces a much smaller denominator and a flatteringly inflated score.

Explanation: ETS beats the benchmark by roughly 11 percent on MASE. Because the denominator is a property of the training data rather than of any model, MASE values are comparable across models on this series and across series in a portfolio, which is why forecasting competitions favour it. ets() here selects a multiplicative error, damped trend and multiplicative seasonality specification automatically.

Explanation: Two zeros and one negative mean MAPE is not usable, and the flag comes back 0. R will happily return Inf for the zero rows and a misleading finite number for the negative one rather than erroring, so this check has to be explicit. When it fails, switch to MASE or to a scale-dependent metric reported alongside the series mean.

Explanation: window() subsets by calendar time rather than by position, so the c(year, month) pairs are readable and the resulting objects keep their frequency and start date. Random train/test splits, standard in cross-sectional machine learning, leak future information into training here and produce accuracy figures that collapse in production.

Explanation: The test MAPE of 15.523 and MASE of 2.494 match what you computed by hand in Exercises 3.1 and 3.5, confirming the formulas. Training MASE is exactly 1.000 because the seasonal naive method is its own scaling benchmark. Call accuracy() with only the forecast object and you get the training row alone, which is not evidence of anything.

Explanation: Name-based indexing survives changes in column order, whereas acc[2, 2] breaks silently if a future package version inserts a metric. This one-liner is the building block for the comparison tables in Exercise 4.7 and Section 5, where the same extraction runs once per model.

Explanation: Test error is 2.4 times training error. For a method with zero fitted parameters that gap is not overfitting: it is the cost of forecasting up to 24 months ahead when the training row measures one-step-ahead fit. Comparing a multi-step test row against a one-step training row is the most common misreading of this table.

Explanation: Training RMSE drops from 32.5 to 8.9 while test RMSE only improves from 77.0 to 72.5. A fitted model always looks dramatically better in-sample because it optimised that fit; the honest improvement is the 6 percent on the test row. Theil's U below 1 would mean beating the naive benchmark, and at 1.357 this forecast does not.

Explanation: The search lands on ARIMA(1,1,0)(0,1,0)[12], which differences once at lag 1 and once at lag 12. Training ACF1 of 0.008 means the residuals are essentially uncorrelated, a good sign that Section 6 tests formally. Note the model selection used AICc on training data only, so the test row remains a fair evaluation.

Explanation: Extracting the test row returns a named numeric vector, so a[["RMSE"]] gives a bare number rather than a one-element vector with a stray name. All three metrics agree on the ranking here, which is reassuring; when they disagree, the metric that matches the cost of being wrong in your business is the one that decides.

Explanation: December 1958 prints as 1958.917 because time() expresses months as fractions of a year, eleven twelfths being 0.917. The flag of 1 confirms a clean boundary. An off-by-one window() call that repeats December in both sets inflates test scores in a way no metric will ever reveal on its own, so this assertion belongs in any automated pipeline.

Explanation: The spread is enormous: the mean method is nearly three times worse than seasonal naive, because averaging a strongly trending series produces a forecast below every future value. Wrapping sapply() in as.numeric() drops the inherited names so the data frame gets clean row numbers instead of duplicated labels.

Explanation: which.min() returns the index of the first minimum, so ties resolve to the earliest row rather than erroring. Automating the choice matters when the same script runs across hundreds of series in a nightly job, where no human is available to read a table. Swap the column name to rank on MAE or MASE instead.

Explanation: ETS edges ARIMA by about 2 percent, which is well inside the noise you would expect from a single 24-month test window. A margin this thin is not a reason to declare a permanent winner; the rolling-origin evaluation in Section 7 averages over many origins and gives a far more stable verdict.

Explanation: Here AICc and test RMSE happen to agree, both favouring multiplicative seasonality, which is the right answer for a series whose seasonal swings grow with its level. They agree because the model families are comparable and the sample is honest; AICc is only valid within one model class fitted to identical data, so it can never be used to compare an ETS against an ARIMA on differenced data.

Explanation: The average lands between the two members rather than beating both, which happens when the models make highly correlated errors: both under-forecast the same climbing trend, so averaging cannot cancel anything. Combinations pay off when the members fail in different directions, and checking that empirically takes exactly the three lines above.

Explanation: A 3.6 percent gain is honest but small, and it is the number worth quoting to a stakeholder because it answers "compared to what". Skill scores are bounded above by 100 and unbounded below, so a model twice as bad as the benchmark scores minus 100. Always state which benchmark was used, since the score is meaningless without it.

Explanation: Order matters in the pipe: calling mutate() before arrange() would number the rows in their original order and produce a scrambled rank column. row_number() needs no arguments inside mutate() because it reads the current row position. Note that the MAE column happens to agree with the RMSE ranking here, which is not guaranteed.

Explanation: A mean of -0.016 against a standard deviation of 9.6 is effectively zero, so the fit carries no in-sample bias. A clearly nonzero mean would mean the forecasts can be improved for free by adding a constant. Note these are in-sample one-step residuals, not the multi-step test errors examined in Exercise 6.5, and the two often tell different stories.

Explanation: With p above 0.05 you do not reject independence, so the residuals pass as white noise at the usual threshold, though 0.084 is close enough to be worth watching. The fitdf argument subtracts the number of estimated coefficients from the degrees of freedom: skip it and the test becomes too permissive because it credits the model with fitting for free.

Explanation: Lag 3 at -0.174 is the largest, close to the approximate significance bound of 2 divided by the square root of 120, which is about 0.18. The Ljung-Box test in the previous exercise pools all lags into one verdict, while this vector shows exactly where the structure sits, which is what you need to decide whether adding an MA term is worth trying.

Explanation: A symmetric normal interval would put the two percentiles near plus and minus 18.8, so the empirical interval is noticeably lopsided: shorter on the downside, longer on the upside. That right skew is typical of a series with multiplicative growth and it warns that normal-theory prediction intervals will misstate risk, which Exercise 7.4 confirms with coverage.

Explanation: Every single one of the 24 test errors is positive, so the model under-forecasts the entire held-out period. Contrast this with the near-zero in-sample residual mean from Exercise 6.1: the model fits history without bias yet extrapolates a trend that is too shallow. Only out-of-sample errors expose that failure mode.

Explanation: Both models under-forecast every single test month, so the problem is the data rather than the model class: air travel accelerated in 1959 and 1960 beyond anything in the training window. When every candidate is biased in the same direction, look for a regime change or a missing driver rather than tuning parameters.

Explanation: Test error is nearly eight times the residual spread. Some of that gap is legitimate, because uncertainty compounds with horizon, but a factor of eight also reflects the systematic bias found in Exercise 6.5. Quoting residual standard deviation as though it were forecast accuracy is the most common way a forecast gets oversold internally.

Explanation: tsCV() returns a series of errors aligned to the time the forecast was made, with NA at the start where too little history exists, so na.rm = TRUE is mandatory. Averaging over roughly a hundred origins instead of one makes this estimate far less sensitive to where you happened to cut the series.

Explanation: Error climbs gently from 36.45 at one month to 37.80 at twelve, a much flatter profile than the single-split result in Exercise 2.6 because averaging over many origins washes out the peculiarities of 1959 and 1960. tsCV() needs a function of exactly (x, h), which is why the wrapper exists rather than passing snaive bare.

Explanation: Forty origins produce an RMSE of 136.4, lower than the 168.1 from Exercise 7.1 because starting at observation 60 skips the volatile pre-1900 era entirely. Which origins you include changes the answer, so state the evaluation window whenever you quote a cross-validated number. Growing a vector with c() is fine at this size but preallocate for long series.

Explanation: An interval advertised at 95 percent captures only 54 percent of the actuals, so it is badly overconfident. $lower and $upper are matrices with one column per requested level, hence the [, 1]. Under-coverage this severe usually traces back to the same shallow trend that produced the one-sided bias in Exercise 6.5.

Explanation: An average width of 140 passengers is wide relative to a series averaging around 450, and it still fails to cover. Coverage and sharpness must always be read together: a useless interval spanning zero to infinity has perfect coverage, and a razor-thin one has none. Good intervals hit their nominal rate with the smallest width that achieves it.

Explanation: The 80 percent interval catches 4 percent of observations and the 95 percent interval catches 54 percent, so both are far too narrow and the failure worsens as the level tightens. Because every test error is positive, the actuals sit above the upper bound almost everywhere, which is a bias problem showing up as a coverage problem.

Explanation: All four metrics rank the models identically, which is the comfortable case; when they disagree, decide in advance which one matches the cost of being wrong rather than picking the flattering one afterwards. Reusing scale_ap across every row is what makes the MASE column comparable. Note that even the winner is biased and under-covered, so this table alone is not a green light to ship.