Index Ai Academy
Phase 11 · Data engineering

Columnar files and a local warehouse

Rows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Training data rarely lives in a tidy CSV. It lives in tables with hundreds of columns and billions of rows, and almost every question asked of them touches a handful of those columns: one feature, one label, one week. The file format used for analytics is built around that fact. It stores each column together, compresses it hard, and keeps notes that let a query skip most of the file without reading it. This lesson opens that format up, writes real Parquet files in your browser, and reads their notes to see why the same question can cost a small fraction as much.

What you'll be able to do

  • Say how row storage and column storage differ, and which questions each answers cheaply
  • Explain dictionary and run-length encoding, and why a sorted, repetitive column shrinks
  • Read a Parquet file's row groups and their minimum and maximum values, and predict which ones a query can skip
  • Query files with SQL using DuckDB, and hand the result to pandas
Quick check
A guess before you start. A table has 200 columns and a billion rows. An analyst asks for the average of one column. Which way of laying the table out on disk lets that question be answered while reading the least data?
  1. Either: a file has to be read from start to finish anyway
  2. The rows sorted so that the largest values come first
  3. All the values of each column kept together, one column after another
  4. Each row's 200 values kept together, one row after another
Show the answer

All the values of each column kept together, one column after another

If each column is stored together, the average needs one column's values: a 200th of the table. Stored row by row, the one value wanted from each row sits between 199 that aren't, and the whole file passes through. Analytics formats are laid out by column for this reason.

The idea: store a table by rows, or by columns

This lesson rests on Ratios, rates and proportion (a file that is a quarter the size is a ratio of 1 to 4) and Data in pieces: partitions and mergeable summaries (a dataset kept as files, read in pieces). Two ways of writing a table down decide how much a question has to read.

Rows or columns

There are two ways to write a table down.

Row by row. All of the first order's values, then all of the second's. A CSV file is this, and so is the storage behind the databases that run an application: SQLite, Postgres, MySQL. It suits their work. "Fetch order 4,812 and change its status" touches one whole row.

Column by column. Every order's id, then every order's country, then every order's amount. This is called columnar storage, and it suits the other kind of work: questions about many rows and few columns.

rows = [
    {"order_id": 1, "country": "de", "amount": 20.0},
    {"order_id": 2, "country": "de", "amount": 35.5},
    {"order_id": 3, "country": "fr", "amount": 12.0},
    {"order_id": 4, "country": "us", "amount": 80.0},
]
# the same table, turned on its side
columns = {name: [row[name] for row in rows] for name in rows[0]}
print(columns)

# a row store hands over whole rows
values_read_by_rows = sum(len(row) for row in rows)
# a column store hands over the one column
values_read_by_columns = len(columns["amount"])
print("total amount:", sum(columns["amount"]), "| values read, row store:",
      values_read_by_rows, "| column store:", values_read_by_columns)
It runs right here in your browser, with nothing to install.

The answer is the same, 147.5. The row layout read 12 values to get it and the column layout read 4. With three columns that is a saving of two thirds. With 200 columns it is 199 parts in 200.

Neither layout is better in general. Application databases are row stores because they read and write whole records, one at a time. Analytics systems and training pipelines are column stores because they scan a few columns of everything. The work of moving data from the first kind to the second, and keeping it correct on the way, is a large part of what data engineering means.

Try it now

Step through three questions on the same table. The lit cells are what each one reads.

Why columns shrink

Keeping a column together has a second payoff. A column holds one kind of thing, a country code, a date, a price, and it repeats itself. Repetition is what compression feeds on. Two simple tricks do most of the work.

Dictionary encoding. List each distinct value once, and replace the values with small numbers that point into the list.

Run-length encoding. Where the same value comes several times in a row, write the value once with a count.

# 16 orders, mostly grouped by country
country = ["de"] * 6 + ["fr"] * 3 + ["us"] * 5 + ["de"] * 2

def dictionary_encode(values):
    words = sorted(set(values))
    return words, [words.index(v) for v in values]

def run_length_encode(values):
    runs = []
    for v in values:
        if runs and runs[-1][0] == v:
            runs[-1][1] += 1                 # the same value again: lengthen the run
        else:
            runs.append([v, 1])              # a new value: start a run
    return runs

words, codes = dictionary_encode(country)
print("dictionary:", words)
print("codes:     ", codes)
print("runs:      ", run_length_encode(codes))

shuffled = ["de", "us", "fr", "de", "us", "de", "fr", "us", "de", "de", "us", "fr",
            "de", "us", "de", "de"]
print("runs when the same 16 values are out of order:",
      len(run_length_encode(dictionary_encode(shuffled)[1])))

Sixteen two-letter codes became a dictionary of three words and four runs: "0 six times, 1 three times, 2 five times, 0 twice". The last line holds the same sixteen values in a jumbled order, and they need 14 runs.

That difference matters in practice. A column compresses well when equal values sit next to each other, so how a file is sorted decides how small it gets. And none of this works on rows: a row is a customer id next to a country next to a price, with nothing in common to exploit.

Quick check
What does this print?
def run_length_encode(values):
    runs = []
    for v in values:
        if runs and runs[-1][0] == v:
            runs[-1][1] += 1
        else:
            runs.append([v, 1])
    return runs

print(run_length_encode(["a", "a", "a", "b", "a", "a"]),
      len(run_length_encode(sorted(["a", "a", "a", "b", "a", "a"]))))
  1. [['a', 5], ['b', 1]] 2
  2. [['a', 3], ['b', 1], ['a', 2]] 3
  3. [['a', 3], ['b', 1]] 2
  4. [['a', 3], ['b', 1], ['a', 2]] 2
Show the answer

[['a', 3], ['b', 1], ['a', 2]] 2

A run ends when the value changes, so the two a's after the b start a new run: three runs. Sorted, the five a's sit together and the list needs only two runs. Sorting is what turns scattered repeats into long runs.

In your own words
Why does sorting a file change how well its columns compress?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: encode a column, read a Parquet file, skip row groups

DuckDB runs in the page, so the same SQL builds a table, writes it as CSV and as Parquet, and reads the file's own notes. The cells below do that, and measure what a sorted file saves.

Drill

Undo a run-length encoding

Compression is only useful if nothing is lost. Write run_length_decode(runs): given a list of [value, count] pairs, return the original list of values. Then write runs_saved(values): how many fewer items the encoded form holds, the number of values minus the number of runs that run_length_encode (given) produces.

def run_length_encode(values):
    runs = []
    for v in values:
        if runs and runs[-1][0] == v:
            runs[-1][1] += 1
        else:
            runs.append([v, 1])
    return runs


def run_length_decode(runs):
    return []


def runs_saved(values):
    return 0


print(run_length_decode([["de", 3], ["fr", 1], ["de", 2]]))
print(runs_saved(["de", "de", "fr", "us", "us"]))
In the lesson, your code is checked as you write it.

A real columnar file

Parquet is the columnar file format used across analytics and machine learning. To see one, you need something that reads and writes them. DuckDB is a full analytics database, columnar inside, that runs in the same process as your program the way SQLite does, and here it runs in your browser. The next cell builds a table of 100,000 orders with SQL, writes it out as CSV and as Parquet, and asks both files the same question.

import duckdb, os, time

con = duckdb.connect()
con.execute("""
    create table orders as
    select range as order_id,
           (range * 7919) % 5000 as customer_id,
           ['de', 'fr', 'us', 'gb', 'es'][1 + (range * 31 % 5)] as country,
           round(5 + (range * 37 % 1000) / 10.0, 2) as amount,
           date '2026-01-01' + cast(range // 400 as integer) as day
    from range(100000)
""")
print(con.execute("select * from orders limit 2").fetchall())

con.execute("copy orders to 'orders.csv' (header)")
con.execute("copy orders to 'orders.parquet' (format parquet, row_group_size 10000)")
csv_bytes = os.path.getsize("orders.csv")
parquet_bytes = os.path.getsize("orders.parquet")
print("csv:", csv_bytes, "bytes | parquet:", parquet_bytes, "bytes | parquet is",
      round(parquet_bytes / csv_bytes * 100), "% of the csv")

question = ("select country, count(*), round(sum(amount)) from '{}' "
            "group by country order by country")
start = time.perf_counter()
from_csv = con.execute(question.format("orders.csv")).fetchall()
csv_seconds = time.perf_counter() - start
start = time.perf_counter()
from_parquet = con.execute(question.format("orders.parquet")).fetchall()
parquet_seconds = time.perf_counter() - start
print(from_parquet[:2])
print("same answer from both:", from_csv == from_parquet, "| parquet was faster:",
      parquet_seconds < csv_seconds)

The Parquet file is about a quarter the size of the CSV, holds exactly the same table, and answers faster. The CSV had to be read in full and every number turned from text into a number. From the Parquet file DuckDB took two columns, already stored as numbers and codes.

A Parquet file also describes itself. DuckDB can read that description without reading the data:

import duckdb

con = duckdb.connect()
con.execute("""
    copy (select range as order_id, (range * 7919) % 5000 as customer_id,
                 ['de', 'fr', 'us', 'gb', 'es'][1 + (range * 31 % 5)] as country,
                 round(5 + (range * 37 % 1000) / 10.0, 2) as amount,
                 date '2026-01-01' + cast(range // 400 as integer) as day
          from range(100000))
    to 'orders.parquet' (format parquet, row_group_size 10000)
""")
by_column = con.execute("""
    select path_in_schema, any_value(encodings), sum(total_uncompressed_size),
        sum(total_compressed_size)
    from parquet_metadata('orders.parquet') group by 1 order by 4 desc
""").fetchall()
for name, encoding, raw, stored in by_column:
    print(name.ljust(12), encoding.ljust(17), "| raw", str(raw).rjust(7), "| stored",
          str(stored).rjust(7), "bytes")

Read the last two columns against what each column holds.

  • country has five distinct values in 100,000 rows. It is dictionary encoded and takes almost no space.
  • day changes once every 400 rows, so it is long runs of the same date. It is the smallest of all.
  • amount has a thousand distinct values: a dictionary still helps.
  • order_id is different in every row. There is nothing to share, so it is stored plainly and is the largest column, together with customer_id.

Nearly all of the file is the two id columns. The three columns a report would actually aggregate are a small fraction of it, and a query that asks only for those never reads the rest.

Quick check
A table of website visits has a column browser (6 distinct values), a column visit_id (unique in every row) and a column page_url (about 2,000 distinct pages). In a Parquet file, which column is likely to take the most space?
  1. browser: it is the column with the fewest distinct values
  2. page_url: its values are the longest pieces of text
  3. visit_id: every value is different, so nothing can be shared
  4. All three take the same space: one value per row each
Show the answer

visit_id: every value is different, so nothing can be shared

A dictionary stores each distinct value once. browser needs 6 entries and page_url 2,000, however many million rows there are, so their long text is paid for once. A unique id has as many distinct values as rows, so every one has to be stored.

Skipping what you don't need

Reading only the needed columns is half of what a columnar file offers. The other half is reading only the needed rows, without an index.

A Parquet file is cut into row groups: here, blocks of about ten thousand rows. For every column in every row group, the file records the smallest and the largest value in that block. An engine that is asked for "orders 62,000 to 62,500" can look at those notes first, and skip every block whose range can't contain them.

import duckdb

con = duckdb.connect()
MAKE = """
    select range as order_id, (range * 7919) % 5000 as customer_id,
           ['de', 'fr', 'us', 'gb', 'es'][1 + (range * 31 % 5)] as country,
           round(5 + (range * 37 % 1000) / 10.0, 2) as amount,
           date '2026-01-01' + cast(range // 400 as integer) as day
    from range(100000)
"""
con.execute("copy (" + MAKE
        + ") to 'orders.parquet' (format parquet, row_group_size 10000)")

def ranges(file, column):
    """For each row group: its number, and the column's smallest and largest value."""
    return con.execute(f"""
        select row_group_id, cast(stats_min as bigint), cast(stats_max as bigint)
        from parquet_metadata('{file}') where path_in_schema = '{column}' order by 1
    """).fetchall()

orders = ranges("orders.parquet", "order_id")
for group, low, high in orders[:3]:
    print("row group", group, "holds order_id", low, "to", high)
print("...", len(orders), "row groups in all")
print("groups that can hold orders 62000 to 62500:", [g for g, low,
        high in orders if high >= 62000 and low <= 62500])
print("groups that can hold customer 1234:       ", [g for g, low,
        high in ranges("orders.parquet", "customer_id") if low <= 1234 <= high])

con.execute("copy (select * from (" + MAKE
        + ") order by customer_id) to 'by_customer.parquet' (format parquet, "
          "row_group_size 10000)")
print("the same, in a file sorted by customer:   ", [g for g, low,
        high in ranges("by_customer.parquet", "customer_id") if low <= 1234 <= high])

For the order range, one row group out of ten can hold the answer. The other nine are skipped unread: the query touches a tenth of the file's rows.

For "customer 1234" the notes are no help at all. Customers are scattered through the file, so every row group's range runs from 0 to 4,999 and all ten have to be read. The orders are sorted by order id, and that is the only reason the first lookup was cheap.

The last line writes the same orders sorted by customer. Now each row group covers a narrow band of customer ids, and customer 1234 can only be in one of them. Same data, same format: a tenth of the reading.

This is the practical lesson of the format. A file can be sorted one way, so sort by the column you filter on most. For training data and event logs that is nearly always time.

Quick check
From the lesson on data in pieces: a dataset is stored as one file per day, and a query asks for last week. An engine opens seven files and ignores the rest. Which idea in this lesson is the same idea, inside a single file?
  1. Storing each column's values together and apart from the others
  2. Converting text to numbers once, when the file is written
  3. Dictionary encoding a column that has few distinct values
  4. Skipping row groups whose minimum and maximum can't match the filter
Show the answer

Skipping row groups whose minimum and maximum can't match the filter

Partitioning by day lets an engine skip whole files from their names. Row-group statistics let it skip whole blocks inside a file from their recorded ranges. Both avoid reading data that the filter rules out, and both work only when the data is organized by the thing being filtered on.

Quick check
A Parquet file's row groups record these ranges for the column day_number. A query asks for day_number between 45 and 50. What does this print: the row groups that must be read?
# (row group, smallest, largest)
stats = [(0, 1, 20), (1, 21, 40), (2, 41, 60), (3, 61, 80), (4, 35, 48)]
low, high = 45, 50
print([group for group, smallest,
       largest in stats if largest >= low and smallest <= high])
  1. [2]
  2. [1, 2, 4]
  3. [2, 4]
  4. [0, 1, 2, 3, 4]
Show the answer

[2, 4]

A group must be read if its range overlaps the one asked for. Group 2 holds 41 to 60 and group 4 holds 35 to 48: both can contain a value from 45 to 50. Group 1 stops at 40. Group 4 shows why the test is on overlap and not on order: its range sits out of sequence.

Drill

Which row groups must be read?

Write groups_to_read(stats, low, high). stats is a list of (group, smallest, largest) for one column. Return the list of group numbers whose range can contain a value from low to high, both ends included, in the order they appear. Then write share_skipped(stats, low, high): the share of the groups that can be skipped, rounded to 2 places (0.0 for an empty list of stats).

def groups_to_read(stats, low, high):
    return [group for group, smallest, largest in stats]


def share_skipped(stats, low, high):
    return 0.0


stats = [(0, 0, 10239), (1, 10240, 20479), (2, 20480, 30719), (3, 30720, 40959)]
print(groups_to_read(stats, 20000, 21000), share_skipped(stats, 20000, 21000))
In the lesson, your code is checked as you write it.
Worked example

Worked example: what sorting buys, measured

Put the two halves of the lesson together. Write the same orders twice, once in arrival order and once sorted by country then day, and compare the size of each column and the row groups a filter on country has to read.

import duckdb, os

con = duckdb.connect()
MAKE = """
    select range as order_id, (range * 7919) % 5000 as customer_id,
           ['de', 'fr', 'us', 'gb', 'es'][1 + (range * 31 % 5)] as country,
           round(5 + (range * 37 % 1000) / 10.0, 2) as amount,
           date '2026-01-01' + cast(range // 400 as integer) as day
    from range(100000)
"""
con.execute("copy (" + MAKE
        + ") to 'arrival.parquet' (format parquet, row_group_size 10000)")
con.execute("copy (select * from (" + MAKE
        + ") order by country, day) to 'sorted.parquet' (format parquet, "
          "row_group_size 10000)")

results = []
for file in ("arrival.parquet", "sorted.parquet"):
    meta = (f"select row_group_id, stats_min, stats_max "
            f"from parquet_metadata('{file}') "
            "where path_in_schema = 'country' order by 1")
    groups = con.execute(meta).fetchall()
    to_read = [g for g, low, high in groups if low <= "fr" <= high]
    results.append((file.split('.')[0], len(to_read)))
    total = con.execute(f"select count(*), round(sum(amount)) from '{file}' where "
                        f"country = 'fr'").fetchone()
    print(file.ljust(16), "| row groups to read for country = 'fr':", len(to_read),
          "of", len(groups), "| answer:", total)

import matplotlib.pyplot as plt

plt.bar([name for name, _ in results], [n for _, n in results])
plt.ylabel("row groups read for country = 'fr'")
plt.show()
In the lesson you can change this and run it.

Both files give the same answer, 20,000 French orders, because they hold the same rows. In arrival order every row group contains all five countries, so its range runs from 'de' to 'us' and all ten must be read. Sorted by country, French orders occupy a stretch of neighboring row groups and the rest are skipped by their ranges alone. Nothing about the query changed. The person who wrote the file decided how much the reader would have to read, which is why the layout of stored data is a design decision and not a detail.

In your own words
Why could the engine skip nine row groups for a range of order ids, and what makes a column compress well?

Write a sentence or two, then compare it with a good answer in the lesson.

On the job: Parquet in a pipeline, and a local warehouse

In a week, this lesson is a decision about how a table is stored. Analysts ask why a question takes minutes, and you look at the layout: the sort order, the row groups, the columns the query needs. A training pipeline reads Parquet columns it needs and filters rows it needs. And DuckDB lets you check a few hundred million rows on a laptop before anyone starts a cluster.

A data warehouse is where an organization keeps the data it analyzes: columnar, built for scans over history, and queried with SQL. The large ones, BigQuery, Snowflake and Redshift among them, spread those scans over many machines. DuckDB is the same kind of engine at the scale of one laptop, and that is often enough: a hundred million rows of Parquet is comfortable for it.

Its most useful habit is that it treats files as tables. No loading step, no server:

import duckdb, pandas

con = duckdb.connect()
con.execute("""
    copy (select range as order_id, (range * 7919) % 5000 as customer_id,
                 ['de', 'fr', 'us', 'gb', 'es'][1 + (range * 31 % 5)] as country,
                 round(5 + (range * 37 % 1000) / 10.0, 2) as amount,
                 date '2026-01-01' + cast(range // 400 as integer) as day
          from range(100000))
    to 'orders.parquet' (format parquet)
""")
# a second source, in another format
with open("customers.csv", "w") as f:
    f.write("customer_id,segment\n")
    for c in range(5000):
        f.write(f"{c},{'business' if c % 4 == 0 else 'home'}\n")

top = con.execute("""
    select c.segment, o.country, count(*) as orders, round(sum(o.amount)) as revenue
    from 'orders.parquet' o join 'customers.csv' c using (customer_id)
    where o.day >= date '2026-06-01'
    group by all
    order by revenue desc
    limit 4
""").df()
print(top)
print(type(top).__name__, top.shape)

One query joined a Parquet file to a CSV file, filtered by date, grouped and sorted, and .df() handed the result to pandas as a DataFrame, ready for scikit-learn. This is the everyday shape of preparing training data: SQL does the heavy joining and filtering over files far bigger than memory would hold as a DataFrame, and pandas receives the small result.

Everything from the SQL lessons applies unchanged. What is new is underneath: the engine read three columns of one file, skipped row groups from before June, and never built an index.

Quick check
A team keeps two years of events as Parquet files and needs a daily count of events per country. A colleague proposes loading all of it into a pandas DataFrame first. What is the main advantage of running the SQL in DuckDB over the files instead?
  1. It reads only the columns and row groups the query needs, not everything
  2. It gives a different, more accurate count than pandas would
  3. It makes the files themselves smaller on disk
  4. It removes the need for the data to be stored in Parquet
Show the answer

It reads only the columns and row groups the query needs, not everything

Loading everything into a DataFrame means reading every column of every row into memory before the first calculation. An engine that queries the files directly takes the two columns it needs, skips blocks its filters rule out, and returns only the small result.

In your own words
You are asked to store a year of click events for analysts and for training. What format and sort order would you choose, and why?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 7 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 11 review cards for spaced repetition

Try a quiz question

Which task is a column store better suited to than a row store?

  1. Summing one column across a hundred million rows
  2. Fetching every field of a single customer's record
  3. Updating the status of one order as it ships
  4. Inserting one new row each time a user signs up

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 11 · about 34 hours of study

Professional practice

Design, ship and run real systems, work well in a team, and get hired 43 lessons 1 free
  1. System design fundamentals FreeRequirements, rough numbers, the standard parts and where systems break.40 min
  2. Caches, queues and background jobsSpeed reads with caches, and move slow work to queues that retry safely.40 min
  3. Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
  4. Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
  5. CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
  6. Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
  7. Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
  8. Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
  9. Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
  10. Reliability and incidentsSLOs, error budgets, alerts, on-call and blameless postmortems.40 min
  11. Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
  12. Measuring an AI feature from its logsTyped views over event logs, metric specs, cost from a price table, and rollups you can rerun.40 min
  13. A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
  14. Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
  15. Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
  16. Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
  17. Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
  18. Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
  19. Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
  20. Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
  21. Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
  22. A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
  23. Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
  24. Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
  25. Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
  26. Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
  27. Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
  28. Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
  29. Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
  30. Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
  31. Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
  32. A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
  33. What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
  34. Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
  35. Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
  36. Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
  37. Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
  38. Interviews and your portfolioShow your work, and solve interview problems the way professionals do.40 min
  39. Code reviewWhat to look for, how to give and take feedback, and automating the routine checks.35 min
  40. Reading unfamiliar codeFind your way around a big codebase fast, with tools that map it for you.35 min
  41. Working with AI coding toolsGive agents good context, keep tasks small, and check every change they make.35 min
  42. Capstone: ship a production AI serviceAn AI-backed service with validation, caching, rate limits, fallbacks, logging and evals.45 min
  43. Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write