Index Ai Academy
Glossary · Phase 11 · Professional practice

Data warehouse

Also written: data warehouses

Where an organization keeps the data it analyzes: columnar, built for scans over history, and queried with SQL.

A data warehouse is where an organization keeps the data it analyzes: columnar, built for scans over history, and queried with SQL. The large ones, BigQuery, Snowflake and Redshift among them, spread those scans over many machines. DuckDB is the same kind of engine at the scale of one laptop, and that is often enough: a hundred million rows of Parquet is comfortable for it.

Try it now

Step through three questions on the same table. The lit cells are what each one reads.

Learn it properly

Columnar files and a local warehouse · Rows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB. About 40 minutes.

Other terms from the same lesson

  • columnar storage: Storing a table one column after another, so a query reads only the columns it uses and each column, holding similar values, compresses well.
  • dictionary encoding: Listing each distinct value of a column once and replacing the values with small codes that point into the list.
  • DuckDB: A columnar analytics database that runs inside your program and queries Parquet and CSV files directly with SQL.
  • Parquet: The columnar file format used across analytics and machine learning. It is cut into row groups that record each column's smallest and largest value.
  • row group: A block of rows in a Parquet file, stored column by column with each column's minimum and maximum, so a query can skip the block.
  • run-length encoding: Writing a repeated value once with a count of how many times it comes in a row.
Keep learning

More in plain English.

1286 terms in the glossary, each with the lesson that teaches it.

1286 terms

The glossary

Every word you'll meet in AI and machine learning, explained in everyday language.

Phase 11 · lesson 40 min

Columnar files and a local warehouse

Rows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write