Index Ai Academy
Phase 3 · Data analysis

Forecasting: baselines, lag features and backtests

Three baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

A buyer's question is never "what happened?" It is "how many shoes should I order for the months ahead?" Forecasting answers it, and the way it fails is specific to time: a model that has seen the future looks brilliant in testing and is useless in use. This lesson builds the baselines every forecast has to beat, features that look only backward, and a backtest that scores a model the way it will be used. The same habits apply to forecasting traffic, token spend, and a training run's loss curve.

What you'll be able to do

  • Score a forecast only on dates after its training data, and say why a shuffled split overstates it
  • Build three baselines (last value, a recent mean, the same month last year) and measure MAE against them
  • Make lag and rolling features with shift so that no row sees its own future
  • Run a backtest with TimeSeriesSplit and read a forecast's error range from its past errors
Quick check
A guess first. A shop sold 40 units last December and 55 this November. With no model at all, what is the simplest sensible forecast for this December?
  1. 95, the two added together
  2. 47.5, the average of the two
  3. 40, the same month last year
  4. 55, the latest value
Show the answer

40, the same month last year

Sales have a yearly shape, so December is best compared with December. This rule, the same period last time, is called a seasonal naive forecast, and it is hard to beat.

The idea: a forecast is scored on dates it never saw

Purpose. Predict the months ahead, and score the prediction in the one way that matches how it will be used.

A forecast is a prediction about dates that have not happened yet. To judge one, you pretend: cut the history at a date, build the forecast from the rows before the cut, and compare it with the rows after. The rows after the cut are the test. They must be the later rows. A random split would train on March and test on February, which uses the future to predict the past and flatters every model.

Before any model, set the bar with baselines, forecasts that need no fitting:

  • Last value: every future month equals the latest month.
  • Recent mean: every future month equals the average of the last few.
  • Same month last year: each month repeats its value from 12 months earlier.

A model that cannot beat the third one on a seasonal series has added nothing. The score is MAE, the mean absolute error: the average of how far each forecast is from what happened, in the units of the data.

MAE=1n∑t∣yt−y^t∣\text{MAE} = \frac{1}{n}\sum_{t} \left| y_t - \hat y_t \right|

In plain words: for each test month take the miss, ignore its sign, and average the misses.

Here is a toy with a four-step season, two years long: 10, 20, 30, 20, then 12, 22, 32, 22. The first year trains the forecasts and the second year tests them.

import numpy as np

y = np.array([10, 20, 30, 20, 12, 22, 32, 22])
train, test = y[:4], y[4:]
last_value = np.full(4, train[-1])
print("last value:", last_value, "MAE", np.abs(test - last_value).mean())
print("same step last year:", train, "MAE", np.abs(test - train).mean())
It runs right here in your browser, with nothing to install.
Quick check
From the lesson on time series with pandas: shift moves a series later, so each row holds an earlier value. What does this print?
import pandas as pd

s = pd.Series([10, 20, 30, 20])
print(s.shift(1).tolist(), s.shift(2).tolist())
  1. [20.0, 30.0, 20.0, nan] [30.0, 20.0, nan, nan]
  2. [10, 20, 30, 20] [10, 20, 30, 20]
  3. [nan, 10.0, 20.0, 30.0] [nan, 10.0, 20.0, 30.0]
  4. [nan, 10.0, 20.0, 30.0] [nan, nan, 10.0, 20.0]
Show the answer

[nan, 10.0, 20.0, 30.0] [nan, nan, 10.0, 20.0]

Each value moves down one place for shift(1) and two for shift(2), leaving empty values at the top. A row's shifted value is something known before that row, which is what makes it safe to use as a feature.

The picture. The same toy series, forecast in two ways. The dashed line marks the cut: everything left of it is history, everything right of it is the test. Step through to see each forecast and its misses.

Try it now

A forecast scored only to the right of the cut. The numbers are the ones the toy cell above prints.

In your own words
Why must the test months come after the training months, and not be picked at random?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: baselines, lag features and a backtest

Purpose. Score the three baselines on the shop's real orders, then see whether a model on lag features beats them.

Baselines on the shop's orders

The buyer's monthly shoe orders, from the lesson on time series, run from January 2019 to May 2025: 77 complete months. The last 12, June 2024 to May 2025, are the test. The three baselines need no fitting:

Worked example

Worked example: three baselines, scored on the last 12 months

Cut the history at May 2024. Build each baseline from the training months and score it on the test months with MAE.

import pandas as pd

daily = pd.read_csv("https://api.gradient.test/v1/datasets/shoe-orders.csv")
daily["date"] = pd.to_datetime(daily["date"])
monthly = daily.set_index("date")["orders"].resample("MS").sum().iloc[:-1]

train, test = monthly.loc[:"2024-05"], monthly.loc["2024-06":]
forecasts = {
    "last value": pd.Series(train.iloc[-1], index=test.index),
    "mean of last 3": pd.Series(train.iloc[-3:].mean(), index=test.index),
    "same month last year": monthly.shift(12).loc[test.index],
}
print(len(train), "training months,", len(test), "test months")
for name, f in forecasts.items():
    print(f"{name:22} MAE {(test - f).abs().mean():.1f}")
print("average test month:", round(test.mean(), 1))
In the lesson you can change this and run it.

The first two baselines are off by 195 and 151 orders in an average month of 663. Copying the same month from last year is off by 37, about 6%. The first two ignore the season, so the winter months are forecast far too high and the summer months far too low. monthly.shift(12) puts each month's year-ago value on its own row, and every one of those values comes from before the test starts, so nothing leaks. This is the number any model must beat.

Code flow. The same-month-last-year forecast is a loop over the test months: for each, take the value 12 rows back, and keep the miss. Step through it on the toy series, where the season is 4 steps long, so the lookup is 4 rows back.

Quick check
A forecast is built with monthly.shift(12). Which line of the test months does it never use?
  1. Values from the training months
  2. Values from the previous year
  3. Any value from the test months themselves
  4. Values from 12 months earlier
Show the answer

Any value from the test months themselves

shift(12) pulls each month's value from a year earlier. For June 2024 that is June 2023, which is in the training months. A test month's own value is never an input, so the forecast cannot peek.

Lag features and a model

A model can use more than one past value at once. A lag feature is an earlier value put on the row as a column: lag_1 is last month, lag_12 is the same month last year. A model learns how much weight each deserves. Every feature comes from shift, so a row only ever sees values from before it:

import pandas as pd
from sklearn.linear_model import Ridge

daily = pd.read_csv("https://api.gradient.test/v1/datasets/shoe-orders.csv")
daily["date"] = pd.to_datetime(daily["date"])
monthly = daily.set_index("date")["orders"].resample("MS").sum().iloc[:-1]

feat = pd.DataFrame({"lag_1": monthly.shift(1), "lag_12": monthly.shift(12),
                     "y": monthly}).dropna()
cols = ["lag_1", "lag_12"]
tr, te = feat.loc[:"2024-05"], feat.loc["2024-06":]
model = Ridge(alpha=1.0).fit(tr[cols], tr["y"])
pred = model.predict(te[cols])
print(len(tr), "rows,", dict(zip(cols, model.coef_.round(3).tolist())))
print("ridge MAE", round(float((te["y"] - pred).abs().mean()), 1))
naive = (te["y"] - te["lag_12"]).abs().mean()
print("same month last year MAE", round(float(naive), 1))

dropna() removes the first 12 months, which have no lag_12, so the model trains on 53 rows. It leans mostly on lag_12, the season, with a smaller push from last month's level. On this one test year its MAE, 38.8, is a little worse than the 37.4 of copying last year. That is a real result and an honest one: a model has to beat the baseline, and this one has not yet. Twelve test months is also a small sample, so the next cell asks the same question across several windows.

Quick check
A model is trained on rows with lag_1 and lag_12. What does this print: the lags in a row for May 2025, from a series where each month's value is its month number?
import pandas as pd

m = pd.Series(range(1, 18), index=pd.date_range("2024-01-01", periods=17, freq="MS"))
row = pd.DataFrame({"lag_1": m.shift(1), "lag_12": m.shift(12)}).loc["2025-05-01"]
print(row.tolist())
  1. [16.0, 5.0]
  2. [16.0, 17.0]
  3. [5.0, 16.0]
  4. [17.0, 5.0]
Show the answer

[16.0, 5.0]

May 2025 is the 17th value. lag_1 is the 16th value, 16, and lag_12 is the value 12 rows back, the 5th, which is May 2024. The row's own value, 17, is in neither lag.

Drill

Make lag features

Write lag_features(s) for a Series s. Return a DataFrame with columns lag_1, lag_2 and roll3. roll3 is the mean of the three values before each row (not including the row itself), so a row never sees its own value.

import pandas as pd


def lag_features(s):
    ...
In the lesson, your code is checked as you write it.
Quick check
Which of these is NOT safe as a feature for predicting the orders on day t?
  1. The orders on day t − 1
  2. The orders on day t − 7
  3. The mean of the orders on days t − 3 to t
  4. The mean of the orders on days t − 7 to t − 1
Show the answer

The mean of the orders on days t − 3 to t

The window t − 3 to t ends on day t, so it contains the very value being predicted. The other three use only days before t, which are known when the forecast is made.

A backtest across several windows

One test year is one sample. A backtest repeats the cut at several dates: each time, train on everything before the cut and test on the months right after it. TimeSeriesSplit does this in order, and its training rows always come before its test rows:

import pandas as pd
from sklearn.linear_model import Ridge
from sklearn.model_selection import TimeSeriesSplit

daily = pd.read_csv("https://api.gradient.test/v1/datasets/shoe-orders.csv")
daily["date"] = pd.to_datetime(daily["date"])
monthly = daily.set_index("date")["orders"].resample("MS").sum().iloc[:-1]
feat = pd.DataFrame({"lag_1": monthly.shift(1), "lag_12": monthly.shift(12),
                     "y": monthly}).dropna()
cols = ["lag_1", "lag_12"]

for k, (a, b) in enumerate(TimeSeriesSplit(n_splits=4, test_size=12).split(feat)):
    tr, te = feat.iloc[a], feat.iloc[b]
    ridge = Ridge(alpha=1.0).fit(tr[cols], tr["y"])
    mae_ridge = (te["y"] - ridge.predict(te[cols])).abs().mean()
    mae_naive = (te["y"] - te["lag_12"]).abs().mean()
    start, end = te.index[0].strftime("%Y-%m"), te.index[-1].strftime("%Y-%m")
    print(k, len(tr), "train rows | test", start, "to", end, "| ridge",
          round(float(mae_ridge), 1), "| last year", round(float(mae_naive), 1))

The ridge model beats copying last year in one window of four, the third, with 27.1 against 44.2. In the other three it is level or worse, and the first two have only 17 and 29 training rows. Across windows the simple rule is the safer bet, and the earlier single test year hid that. A model is worth trusting when it beats the baseline in most windows, not in one.

Quick check
TimeSeriesSplit cuts 20 rows into 3 windows of 4 test rows each. What does this print: for each window, its number of training rows and the first test row?
import numpy as np
from sklearn.model_selection import TimeSeriesSplit

splits = TimeSeriesSplit(n_splits=3, test_size=4).split(np.arange(20))
print([(len(a), int(b[0])) for a, b in splits])
  1. [(8, 8), (8, 12), (8, 16)]
  2. [(8, 8), (12, 12), (16, 16)]
  3. [(16, 16), (12, 12), (8, 8)]
  4. [(4, 4), (8, 8), (12, 12)]
Show the answer

[(8, 8), (12, 12), (16, 16)]

The test windows are the last 12 rows, four at a time. Each training set is every row before its test window, so it grows: 8, then 12, then 16 rows. A test window always starts where its training set ends.

A forecast as a picture. Plot the test year: the actual orders, the model's forecast, and the same-month-last-year baseline.

import matplotlib.pyplot as plt
import pandas as pd
from sklearn.linear_model import Ridge

daily = pd.read_csv("https://api.gradient.test/v1/datasets/shoe-orders.csv")
daily["date"] = pd.to_datetime(daily["date"])
monthly = daily.set_index("date")["orders"].resample("MS").sum().iloc[:-1]
feat = pd.DataFrame({"lag_1": monthly.shift(1), "lag_12": monthly.shift(12),
                     "y": monthly}).dropna()
cols = ["lag_1", "lag_12"]
tr, te = feat.loc[:"2024-05"], feat.loc["2024-06":]
pred = Ridge(alpha=1.0).fit(tr[cols], tr["y"]).predict(te[cols])

plt.plot(te.index, te["y"], marker="o", label="actual")
plt.plot(te.index, pred, marker="s", label="ridge on lags")
plt.plot(te.index, te["lag_12"], linestyle="--", label="last year")
plt.ylabel("orders per month")
plt.legend()
plt.show()

Both forecasts follow the season. The series grew, so a rule that copies last year is too low in most months.

Drill

Find the bug: a feature that has seen the answer

roll3_feature(s) should give each row the mean of the three values before it, to use as a feature for predicting that row. Its test score looks suspiciously good, because each row's own value is inside its mean. Fix it.

import pandas as pd


def roll3_feature(s):
    return s.rolling(3).mean()
In the lesson, your code is checked as you write it.
In your own words
How can a feature accidentally see the answer, and what do you change to fix it?

Write a sentence or two, then compare it with a good answer in the lesson.

In practice: statsmodels, the 2020 shock and honest ranges

Purpose. Use the standard tools for seasonal series, handle a one-off shock, and give a range as well as a number.

statsmodels: decomposition and exponential smoothing

For a series with a yearly shape, statsmodels offers two classic tools. seasonal_decompose splits a series into a trend, a seasonal pattern and what is left. ExponentialSmoothing forecasts from a level, a trend and a season, each updated as new months arrive. Both are baselines worth running before any lag-feature model:

import pandas as pd
from statsmodels.tsa.holtwinters import ExponentialSmoothing
from statsmodels.tsa.seasonal import seasonal_decompose

daily = pd.read_csv("https://api.gradient.test/v1/datasets/shoe-orders.csv")
daily["date"] = pd.to_datetime(daily["date"])
monthly = daily.set_index("date")["orders"].resample("MS").sum().iloc[:-1]
train, test = monthly.loc[:"2024-05"], monthly.loc["2024-06":]

season = seasonal_decompose(train, model="additive", period=12).seasonal.iloc[:12]
print("seasonal shape, January to December:", season.round(0).astype(int).tolist())

model = ExponentialSmoothing(train, trend="add", seasonal="add", seasonal_periods=12)
fit = model.fit()
miss = (test - fit.forecast(12)).abs().mean()
print("Holt-Winters MAE", round(float(miss), 1))

The seasonal shape is the set of 12 adjustments that the series repeats each year: negative in winter and positive in summer. Holt-Winters forecasts the next 12 months by carrying the level and trend forward and adding that shape. Its MAE on the test year, 33.2, beats both the 37.4 of copying last year and the lag-feature model, with no feature work.

A one-off shock, and a range around the number

In April 2020 orders fell by more than half. The series recovered within months, but a rule that copies "the same month last year" copies the crash into April 2021: a forecast of 233 for a month that sold 608. A model fitted on all months learns from it too. Here is the damage, and a range built from errors the baseline actually made on the training months:

import pandas as pd

daily = pd.read_csv("https://api.gradient.test/v1/datasets/shoe-orders.csv")
daily["date"] = pd.to_datetime(daily["date"])
monthly = daily.set_index("date")["orders"].resample("MS").sum().iloc[:-1]
train, test = monthly.loc[:"2024-05"], monthly.loc["2024-06":]

april = monthly.shift(12).loc["2021-04-01"], monthly.loc["2021-04-01"]
print("April 2021: copy of April 2020 =", int(april[0]), "actual =", int(april[1]))

past = (train - train.shift(12)).dropna()
low, high = past.quantile([0.1, 0.9])
forecast = monthly.shift(12).loc[test.index]
inside = ((test >= forecast + low) & (test <= forecast + high)).mean()
print("80% range from past errors:", round(float(low)), "to", round(float(high)))
print("share of test months inside the range:", round(float(inside), 2))

A forecast is more useful as a range: "between 80 below and 90 above last year's figure" lets the buyer order for the likely case and keep a margin for the rest. The range comes from the past misses, and the last line checks it on months it was not built from. A range that holds far less than its stated 80% share is too narrow, and a range that holds all of them is too wide.

Where this turns up in an engineer's week. A capacity plan for next quarter's traffic, a token-spend forecast for a finance review, an alert that fires when a metric leaves its usual range, a check on whether a training run's loss is on track. Each starts with the same two numbers: the best baseline's error and the model's error on later dates. Each also needs a decision about shocks, such as a launch, an outage or a holiday, that should be left out of what the model learns from. When the baseline is hard to beat, report that: a simple rule that is close enough is cheaper to run and easier to explain.

On the job: write the baseline's error in the first line of any forecasting report. A model's number means nothing until it sits beside it.

Goes deeper in

In your own words
Why report a range with a forecast, and how can you build one from the past?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 7 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 10 review cards for spaced repetition

Try a quiz question

A forecast scores an MAE of 40 on a series whose average month is 660. The same-month-last-year baseline scores 37. What does that say?

  1. The forecast has not beaten a rule that needs no model
  2. The forecast is 40% accurate on this series
  3. The baseline is broken and should be discarded
  4. MAE cannot be used to compare two forecasts

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 3 · about 39 hours of study

Machine learning

Learning from data, and knowing when it worked 50 lessons 1 free
  1. What machine learning is FreePrograms that learn rules from examples instead of being told them.30 min
  2. Features and labelsSupervised learning: inputs, targets and predictions.30 min
  3. Train, validation and test setsHold data back so you can trust your results.38 min
  4. Preparing dataClean, encode and scale data before a model sees it.40 min
  5. Linear regressionFit a line, and predict numbers from features.42 min
  6. Loss functionsOne number that says how wrong the model is.35 min
  7. Training with gradient descentAdjust weights step by step to reduce the loss.42 min
  8. Learning rateStep size, convergence and divergence.35 min
  9. Logistic regressionPredict probabilities for yes/no outcomes.42 min
  10. Multiclass classificationSoftmax and cross-entropy for more than two classes.42 min
  11. The same three models in PyTorchA line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.35 min
  12. Precision, recall and F1Accuracy lies; these don't.42 min
  13. ROC curves and AUCJudge a classifier across every threshold.30 min
  14. Calibration: can the probability be trusted?Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.30 min
  15. Overfitting and underfittingMemorizing versus learning.30 min
  16. RegularizationPenalize complexity so models generalize.40 min
  17. Bias and varianceThe two ways a model can be wrong.30 min
  18. Cross-validationReuse limited data for more reliable estimates.42 min
  19. Imbalanced classes in depthRare classes measured: why rebalancing mostly moves the threshold, correcting inflated probabilities, average precision against AUC, why the count of rare cases matters, and thresholds from costs.45 min
  20. Naive Bayes: classifying with countsA classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.45 min
  21. Nearest neighborsPredict from the most similar examples.35 min
  22. Decision treesLearn a flowchart of yes/no questions.42 min
  23. Random forests and boostingMany weak models voting beat one strong one.42 min
  24. Gradient boosting in depthBoosting built from scratch: trees fit to the gradient of the loss, depth and interactions, learning rate against number of trees, early stopping, and what XGBoost and LightGBM add.45 min
  25. Support vector machinesThe widest margin, support vectors, the hinge loss and C, kernels for curved boundaries, and when an SVM fits.40 min
  26. Recommender systemsShrunk averages, item-to-item collaborative filtering, matrix factorization, and cold start.40 min
  27. Project: a price modelMultiple regression on log prices, one-hot categories, percentage effects and an extrapolation flag.45 min
  28. Feature engineeringTurn raw data into signals a model can use.35 min
  29. The scikit-learn workflowFit, predict and evaluate with the standard ML library.45 min
  30. Explaining a model: what it uses, and why this predictionPermutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.35 min
  31. Capstone: churn predictionPredict which customers will leave, and measure it properly.45 min
  32. ClusteringGroup similar items without labels (k-means).35 min
  33. Clusters of any shape: DBSCAN and hierarchiesWhy k-means needs round clusters, DBSCAN's core points and noise, and hierarchical clustering with its linkages.40 min
  34. Anomaly detectionMasking, robust scores with the median and MAD, baselines from the past, isolation forests and thresholds from a review budget.40 min
  35. Dimensionality reductionVisualize and compress high-dimensional data.35 min
  36. Cleaning messy dataDuplicates, junk rows, text numbers, mixed dates and missing values that mean something.45 min
  37. Grouping, pivoting and reshapingNamed aggregation, pivot tables, transform, running totals and groups too small to rank.45 min
  38. Data in pieces: partitions and mergeable summariesSplit, summarize and combine; which statistics merge exactly, merging variances, sketches for the rest, and partitioning well.30 min
  39. Combining tables with mergeJoin on keys safely, stack tables, and divide by the right denominator.44 min
  40. Time series with pandasResample, smooth, compare across years, and test whether one series leads another.45 min
  41. Forecasting: baselines, lag features and backtestsThree baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.40 min
  42. Text mining: TF-IDF, search and labelsBag of words and TF-IDF, similarity search and its blind spots, text classifiers and topics, and when LLM labels are worth it.35 min
  43. Location dataHaversine distances, nearest places, GeoJSON, point in polygon and simple maps.40 min
  44. Comparing groups: t-testsWelch's t-test, pooled vs average rates, and a control group for before-and-after changes.40 min
  45. More tests: groups, categories, relationshipsChoose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.40 min
  46. Project: an exploratory analysis reportFive business questions answered with cleaning, joins, charts and a test, then written up.45 min
  47. Bandits: explore or exploitLearning from rewards with no labels: a running estimate for each option, why the greedy choice gets stuck, epsilon-greedy and an optimism bonus.35 min
  48. Decisions in sequence: states, rewards and valueA Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy.40 min
  49. Q-learning: learning values by doingA value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.40 min
  50. Policy gradients: learning what to do directlyA policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.40 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write