Forecasting: baselines, lag features and backtests
Three baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.
What it covers6
Why it matters
A buyer's question is never "what happened?" It is "how many shoes should I order for the months ahead?" Forecasting answers it, and the way it fails is specific to time: a model that has seen the future looks brilliant in testing and is useless in use. This lesson builds the baselines every forecast has to beat, features that look only backward, and a backtest that scores a model the way it will be used. The same habits apply to forecasting traffic, token spend, and a training run's loss curve.
What you'll be able to do
- Score a forecast only on dates after its training data, and say why a shuffled split overstates it
- Build three baselines (last value, a recent mean, the same month last year) and measure MAE against them
- Make lag and rolling features with shift so that no row sees its own future
- Run a backtest with TimeSeriesSplit and read a forecast's error range from its past errors
Show the answer
40, the same month last year
Sales have a yearly shape, so December is best compared with December. This rule, the same period last time, is called a seasonal naive forecast, and it is hard to beat.
The idea: a forecast is scored on dates it never saw
Purpose. Predict the months ahead, and score the prediction in the one way that matches how it will be used.
A forecast is a prediction about dates that have not happened yet. To judge one, you pretend: cut the history at a date, build the forecast from the rows before the cut, and compare it with the rows after. The rows after the cut are the test. They must be the later rows. A random split would train on March and test on February, which uses the future to predict the past and flatters every model.
Before any model, set the bar with baselines, forecasts that need no fitting:
- Last value: every future month equals the latest month.
- Recent mean: every future month equals the average of the last few.
- Same month last year: each month repeats its value from 12 months earlier.
A model that cannot beat the third one on a seasonal series has added nothing. The score is MAE, the mean absolute error: the average of how far each forecast is from what happened, in the units of the data.
In plain words: for each test month take the miss, ignore its sign, and average the misses.
Here is a toy with a four-step season, two years long: 10, 20, 30, 20, then 12, 22, 32, 22. The first year trains the forecasts and the second year tests them.
import numpy as np y = np.array([10, 20, 30, 20, 12, 22, 32, 22]) train, test = y[:4], y[4:] last_value = np.full(4, train[-1]) print("last value:", last_value, "MAE", np.abs(test - last_value).mean()) print("same step last year:", train, "MAE", np.abs(test - train).mean())
shift moves a series later, so each row holds an earlier value. What does this print?import pandas as pd s = pd.Series([10, 20, 30, 20]) print(s.shift(1).tolist(), s.shift(2).tolist())
Show the answer
[nan, 10.0, 20.0, 30.0] [nan, nan, 10.0, 20.0]
Each value moves down one place for shift(1) and two for shift(2), leaving empty values at the top. A row's shifted value is something known before that row, which is what makes it safe to use as a feature.
The picture. The same toy series, forecast in two ways. The dashed line marks the cut: everything left of it is history, everything right of it is the test. Step through to see each forecast and its misses.
A forecast scored only to the right of the cut. The numbers are the ones the toy cell above prints.
Write a sentence or two, then compare it with a good answer in the lesson.
In code: baselines, lag features and a backtest
Purpose. Score the three baselines on the shop's real orders, then see whether a model on lag features beats them.
Baselines on the shop's orders
The buyer's monthly shoe orders, from the lesson on time series, run from January 2019 to May 2025: 77 complete months. The last 12, June 2024 to May 2025, are the test. The three baselines need no fitting:
Worked example: three baselines, scored on the last 12 months
Cut the history at May 2024. Build each baseline from the training months and score it on the test months with MAE.
import pandas as pd daily = pd.read_csv("https://api.gradient.test/v1/datasets/shoe-orders.csv") daily["date"] = pd.to_datetime(daily["date"]) monthly = daily.set_index("date")["orders"].resample("MS").sum().iloc[:-1] train, test = monthly.loc[:"2024-05"], monthly.loc["2024-06":] forecasts = { "last value": pd.Series(train.iloc[-1], index=test.index), "mean of last 3": pd.Series(train.iloc[-3:].mean(), index=test.index), "same month last year": monthly.shift(12).loc[test.index], } print(len(train), "training months,", len(test), "test months") for name, f in forecasts.items(): print(f"{name:22} MAE {(test - f).abs().mean():.1f}") print("average test month:", round(test.mean(), 1))
The first two baselines are off by 195 and 151 orders in an average month of 663. Copying the same month from last year is off by 37, about 6%. The first two ignore the season, so the winter months are forecast far too high and the summer months far too low. monthly.shift(12) puts each month's year-ago value on its own row, and every one of those values comes from before the test starts, so nothing leaks. This is the number any model must beat.
Code flow. The same-month-last-year forecast is a loop over the test months: for each, take the value 12 rows back, and keep the miss. Step through it on the toy series, where the season is 4 steps long, so the lookup is 4 rows back.
monthly.shift(12). Which line of the test months does it never use?Show the answer
Any value from the test months themselves
shift(12) pulls each month's value from a year earlier. For June 2024 that is June 2023, which is in the training months. A test month's own value is never an input, so the forecast cannot peek.
Lag features and a model
A model can use more than one past value at once. A lag feature is an earlier value put on the row as a column: lag_1 is last month, lag_12 is the same month last year. A model learns how much weight each deserves. Every feature comes from shift, so a row only ever sees values from before it:
import pandas as pd from sklearn.linear_model import Ridge daily = pd.read_csv("https://api.gradient.test/v1/datasets/shoe-orders.csv") daily["date"] = pd.to_datetime(daily["date"]) monthly = daily.set_index("date")["orders"].resample("MS").sum().iloc[:-1] feat = pd.DataFrame({"lag_1": monthly.shift(1), "lag_12": monthly.shift(12), "y": monthly}).dropna() cols = ["lag_1", "lag_12"] tr, te = feat.loc[:"2024-05"], feat.loc["2024-06":] model = Ridge(alpha=1.0).fit(tr[cols], tr["y"]) pred = model.predict(te[cols]) print(len(tr), "rows,", dict(zip(cols, model.coef_.round(3).tolist()))) print("ridge MAE", round(float((te["y"] - pred).abs().mean()), 1)) naive = (te["y"] - te["lag_12"]).abs().mean() print("same month last year MAE", round(float(naive), 1))
dropna() removes the first 12 months, which have no lag_12, so the model trains on 53 rows. It leans mostly on lag_12, the season, with a smaller push from last month's level. On this one test year its MAE, 38.8, is a little worse than the 37.4 of copying last year. That is a real result and an honest one: a model has to beat the baseline, and this one has not yet. Twelve test months is also a small sample, so the next cell asks the same question across several windows.
lag_1 and lag_12. What does this print: the lags in a row for May 2025, from a series where each month's value is its month number?import pandas as pd m = pd.Series(range(1, 18), index=pd.date_range("2024-01-01", periods=17, freq="MS")) row = pd.DataFrame({"lag_1": m.shift(1), "lag_12": m.shift(12)}).loc["2025-05-01"] print(row.tolist())
Show the answer
[16.0, 5.0]
May 2025 is the 17th value. lag_1 is the 16th value, 16, and lag_12 is the value 12 rows back, the 5th, which is May 2024. The row's own value, 17, is in neither lag.
Make lag features
Write lag_features(s) for a Series s. Return a DataFrame with columns lag_1, lag_2 and roll3. roll3 is the mean of the three values before each row (not including the row itself), so a row never sees its own value.
import pandas as pd def lag_features(s): ...
Show the answer
The mean of the orders on days t − 3 to t
The window t − 3 to t ends on day t, so it contains the very value being predicted. The other three use only days before t, which are known when the forecast is made.
A backtest across several windows
One test year is one sample. A backtest repeats the cut at several dates: each time, train on everything before the cut and test on the months right after it. TimeSeriesSplit does this in order, and its training rows always come before its test rows:
import pandas as pd from sklearn.linear_model import Ridge from sklearn.model_selection import TimeSeriesSplit daily = pd.read_csv("https://api.gradient.test/v1/datasets/shoe-orders.csv") daily["date"] = pd.to_datetime(daily["date"]) monthly = daily.set_index("date")["orders"].resample("MS").sum().iloc[:-1] feat = pd.DataFrame({"lag_1": monthly.shift(1), "lag_12": monthly.shift(12), "y": monthly}).dropna() cols = ["lag_1", "lag_12"] for k, (a, b) in enumerate(TimeSeriesSplit(n_splits=4, test_size=12).split(feat)): tr, te = feat.iloc[a], feat.iloc[b] ridge = Ridge(alpha=1.0).fit(tr[cols], tr["y"]) mae_ridge = (te["y"] - ridge.predict(te[cols])).abs().mean() mae_naive = (te["y"] - te["lag_12"]).abs().mean() start, end = te.index[0].strftime("%Y-%m"), te.index[-1].strftime("%Y-%m") print(k, len(tr), "train rows | test", start, "to", end, "| ridge", round(float(mae_ridge), 1), "| last year", round(float(mae_naive), 1))
The ridge model beats copying last year in one window of four, the third, with 27.1 against 44.2. In the other three it is level or worse, and the first two have only 17 and 29 training rows. Across windows the simple rule is the safer bet, and the earlier single test year hid that. A model is worth trusting when it beats the baseline in most windows, not in one.
TimeSeriesSplit cuts 20 rows into 3 windows of 4 test rows each. What does this print: for each window, its number of training rows and the first test row?import numpy as np from sklearn.model_selection import TimeSeriesSplit splits = TimeSeriesSplit(n_splits=3, test_size=4).split(np.arange(20)) print([(len(a), int(b[0])) for a, b in splits])
Show the answer
[(8, 8), (12, 12), (16, 16)]
The test windows are the last 12 rows, four at a time. Each training set is every row before its test window, so it grows: 8, then 12, then 16 rows. A test window always starts where its training set ends.
A forecast as a picture. Plot the test year: the actual orders, the model's forecast, and the same-month-last-year baseline.
import matplotlib.pyplot as plt import pandas as pd from sklearn.linear_model import Ridge daily = pd.read_csv("https://api.gradient.test/v1/datasets/shoe-orders.csv") daily["date"] = pd.to_datetime(daily["date"]) monthly = daily.set_index("date")["orders"].resample("MS").sum().iloc[:-1] feat = pd.DataFrame({"lag_1": monthly.shift(1), "lag_12": monthly.shift(12), "y": monthly}).dropna() cols = ["lag_1", "lag_12"] tr, te = feat.loc[:"2024-05"], feat.loc["2024-06":] pred = Ridge(alpha=1.0).fit(tr[cols], tr["y"]).predict(te[cols]) plt.plot(te.index, te["y"], marker="o", label="actual") plt.plot(te.index, pred, marker="s", label="ridge on lags") plt.plot(te.index, te["lag_12"], linestyle="--", label="last year") plt.ylabel("orders per month") plt.legend() plt.show()
Both forecasts follow the season. The series grew, so a rule that copies last year is too low in most months.
Find the bug: a feature that has seen the answer
roll3_feature(s) should give each row the mean of the three values before it, to use as a feature for predicting that row. Its test score looks suspiciously good, because each row's own value is inside its mean. Fix it.
import pandas as pd def roll3_feature(s): return s.rolling(3).mean()
Write a sentence or two, then compare it with a good answer in the lesson.
In practice: statsmodels, the 2020 shock and honest ranges
Purpose. Use the standard tools for seasonal series, handle a one-off shock, and give a range as well as a number.
statsmodels: decomposition and exponential smoothing
For a series with a yearly shape, statsmodels offers two classic tools. seasonal_decompose splits a series into a trend, a seasonal pattern and what is left. ExponentialSmoothing forecasts from a level, a trend and a season, each updated as new months arrive. Both are baselines worth running before any lag-feature model:
import pandas as pd from statsmodels.tsa.holtwinters import ExponentialSmoothing from statsmodels.tsa.seasonal import seasonal_decompose daily = pd.read_csv("https://api.gradient.test/v1/datasets/shoe-orders.csv") daily["date"] = pd.to_datetime(daily["date"]) monthly = daily.set_index("date")["orders"].resample("MS").sum().iloc[:-1] train, test = monthly.loc[:"2024-05"], monthly.loc["2024-06":] season = seasonal_decompose(train, model="additive", period=12).seasonal.iloc[:12] print("seasonal shape, January to December:", season.round(0).astype(int).tolist()) model = ExponentialSmoothing(train, trend="add", seasonal="add", seasonal_periods=12) fit = model.fit() miss = (test - fit.forecast(12)).abs().mean() print("Holt-Winters MAE", round(float(miss), 1))
The seasonal shape is the set of 12 adjustments that the series repeats each year: negative in winter and positive in summer. Holt-Winters forecasts the next 12 months by carrying the level and trend forward and adding that shape. Its MAE on the test year, 33.2, beats both the 37.4 of copying last year and the lag-feature model, with no feature work.
A one-off shock, and a range around the number
In April 2020 orders fell by more than half. The series recovered within months, but a rule that copies "the same month last year" copies the crash into April 2021: a forecast of 233 for a month that sold 608. A model fitted on all months learns from it too. Here is the damage, and a range built from errors the baseline actually made on the training months:
import pandas as pd daily = pd.read_csv("https://api.gradient.test/v1/datasets/shoe-orders.csv") daily["date"] = pd.to_datetime(daily["date"]) monthly = daily.set_index("date")["orders"].resample("MS").sum().iloc[:-1] train, test = monthly.loc[:"2024-05"], monthly.loc["2024-06":] april = monthly.shift(12).loc["2021-04-01"], monthly.loc["2021-04-01"] print("April 2021: copy of April 2020 =", int(april[0]), "actual =", int(april[1])) past = (train - train.shift(12)).dropna() low, high = past.quantile([0.1, 0.9]) forecast = monthly.shift(12).loc[test.index] inside = ((test >= forecast + low) & (test <= forecast + high)).mean() print("80% range from past errors:", round(float(low)), "to", round(float(high))) print("share of test months inside the range:", round(float(inside), 2))
A forecast is more useful as a range: "between 80 below and 90 above last year's figure" lets the buyer order for the likely case and keep a margin for the rest. The range comes from the past misses, and the last line checks it on months it was not built from. A range that holds far less than its stated 80% share is too narrow, and a range that holds all of them is too wide.
Where this turns up in an engineer's week. A capacity plan for next quarter's traffic, a token-spend forecast for a finance review, an alert that fires when a metric leaves its usual range, a check on whether a training run's loss is on track. Each starts with the same two numbers: the best baseline's error and the model's error on later dates. Each also needs a decision about shocks, such as a launch, an outage or a holiday, that should be left out of what the model learns from. When the baseline is hard to beat, report that: a simple rule that is close enough is cheaper to run and easier to explain.
On the job: write the baseline's error in the first line of any forecasting report. A model's number means nothing until it sits beside it.
Goes deeper in
- Cross-validation: other splitters, and why a shuffled fold is wrong for dates.
- Regularization: the ridge penalty used in the lag-feature model.
- Anomaly detection: ranges as alerts.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 7 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 10 review cards for spaced repetition
Try a quiz question
A forecast scores an MAE of 40 on a series whose average month is 660. The same-month-last-year baseline scores 37. What does that say?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 3 · about 39 hours of study
Machine learning
Learning from data, and knowing when it worked
50 lessons 1 free
- What machine learning is FreePrograms that learn rules from examples instead of being told them.30 min
- Features and labelsSupervised learning: inputs, targets and predictions.30 min
- Train, validation and test setsHold data back so you can trust your results.38 min
- Preparing dataClean, encode and scale data before a model sees it.40 min
- Linear regressionFit a line, and predict numbers from features.42 min
- Loss functionsOne number that says how wrong the model is.35 min
- Training with gradient descentAdjust weights step by step to reduce the loss.42 min
- Learning rateStep size, convergence and divergence.35 min
- Logistic regressionPredict probabilities for yes/no outcomes.42 min
- Multiclass classificationSoftmax and cross-entropy for more than two classes.42 min
- The same three models in PyTorchA line, logistic regression and softmax regression fitted both ways: the loss, the gradient and the update opened up, with weight decay and mini-batches.35 min
- Precision, recall and F1Accuracy lies; these don't.42 min
- ROC curves and AUCJudge a classifier across every threshold.30 min
- Calibration: can the probability be trusted?Bands of predicted risk against what happened, the Brier score, why naive Bayes is over-confident, and fixing it on rows the model did not fit.30 min
- Overfitting and underfittingMemorizing versus learning.30 min
- RegularizationPenalize complexity so models generalize.40 min
- Bias and varianceThe two ways a model can be wrong.30 min
- Cross-validationReuse limited data for more reliable estimates.42 min
- Imbalanced classes in depthRare classes measured: why rebalancing mostly moves the threshold, correcting inflated probabilities, average precision against AUC, why the count of rare cases matters, and thresholds from costs.45 min
- Naive Bayes: classifying with countsA classifier trained by counting: Bayes' rule over words, the naive assumption, log scores, smoothing for unseen words, why it is overconfident, and when it beats logistic regression.45 min
- Nearest neighborsPredict from the most similar examples.35 min
- Decision treesLearn a flowchart of yes/no questions.42 min
- Random forests and boostingMany weak models voting beat one strong one.42 min
- Gradient boosting in depthBoosting built from scratch: trees fit to the gradient of the loss, depth and interactions, learning rate against number of trees, early stopping, and what XGBoost and LightGBM add.45 min
- Support vector machinesThe widest margin, support vectors, the hinge loss and C, kernels for curved boundaries, and when an SVM fits.40 min
- Recommender systemsShrunk averages, item-to-item collaborative filtering, matrix factorization, and cold start.40 min
- Project: a price modelMultiple regression on log prices, one-hot categories, percentage effects and an extrapolation flag.45 min
- Feature engineeringTurn raw data into signals a model can use.35 min
- The scikit-learn workflowFit, predict and evaluate with the standard ML library.45 min
- Explaining a model: what it uses, and why this predictionPermutation importance on held-back rows, a linear model's contributions for one customer, partial dependence, and what none of them can say.35 min
- Capstone: churn predictionPredict which customers will leave, and measure it properly.45 min
- ClusteringGroup similar items without labels (k-means).35 min
- Clusters of any shape: DBSCAN and hierarchiesWhy k-means needs round clusters, DBSCAN's core points and noise, and hierarchical clustering with its linkages.40 min
- Anomaly detectionMasking, robust scores with the median and MAD, baselines from the past, isolation forests and thresholds from a review budget.40 min
- Dimensionality reductionVisualize and compress high-dimensional data.35 min
- Cleaning messy dataDuplicates, junk rows, text numbers, mixed dates and missing values that mean something.45 min
- Grouping, pivoting and reshapingNamed aggregation, pivot tables, transform, running totals and groups too small to rank.45 min
- Data in pieces: partitions and mergeable summariesSplit, summarize and combine; which statistics merge exactly, merging variances, sketches for the rest, and partitioning well.30 min
- Combining tables with mergeJoin on keys safely, stack tables, and divide by the right denominator.44 min
- Time series with pandasResample, smooth, compare across years, and test whether one series leads another.45 min
- Forecasting: baselines, lag features and backtestsThree baselines, lag features that never see their own future, a walk-forward backtest, and a range around the number.40 min
- Text mining: TF-IDF, search and labelsBag of words and TF-IDF, similarity search and its blind spots, text classifiers and topics, and when LLM labels are worth it.35 min
- Location dataHaversine distances, nearest places, GeoJSON, point in polygon and simple maps.40 min
- Comparing groups: t-testsWelch's t-test, pooled vs average rates, and a control group for before-and-after changes.40 min
- More tests: groups, categories, relationshipsChoose a test, ANOVA by hand, chi-square on counts, Pearson vs Spearman, one-sided tests.40 min
- Project: an exploratory analysis reportFive business questions answered with cleaning, joins, charts and a test, then written up.45 min
- Bandits: explore or exploitLearning from rewards with no labels: a running estimate for each option, why the greedy choice gets stuck, epsilon-greedy and an optimism bonus.35 min
- Decisions in sequence: states, rewards and valueA Markov decision process on a grid world: discounted return, the value of a state, the Bellman equation, and value iteration to the best policy.40 min
- Q-learning: learning values by doingA value for every action, learned from experience with no rules given: the update, a grid world you can watch, and why optimism explores better than chance.40 min
- Policy gradients: learning what to do directlyA policy as probabilities you can nudge: turn up what worked, measure it against a baseline, and see how the same update trains a language model.40 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write