CI/CD pipelines
Test every change automatically, and ship with gates, flags and rollbacks.
Create a free account Try part of it here
Why it matters
On a professional team, nobody's change reaches users until a machine has built it, tested it and checked it, every time. That machine is the CI/CD pipeline. It catches mistakes in minutes instead of in production, lets many people change one codebase safely, and makes shipping routine rather than scary. Teams that deploy many times a day aren't braver; they have better pipelines.
What you'll be able to do
- Explain continuous integration, delivery and deployment, and what each one guarantees
- Read and write a GitHub Actions workflow with jobs, dependencies, a matrix and secrets
- Plan a pipeline as stages of jobs from their dependencies, and skip jobs whose inputs failed
- Separate deploying from releasing with feature flags and stable percentage rollouts
What it covers
- Three terms
- Which job runs when?
- Designing a good pipeline
- Why fast checks go first
- Feature flags: deploy is not release
- Stable buckets
- Build once
Three terms
Picture a car factory. Each part gets checked as it's fitted, finished cars wait in the lot until someone orders them released, and at some factories they roll straight out to dealers. Software has the same three stages:
- Continuous integration (CI): every push and pull request runs the checks: formatting, linting (automatic checks for suspicious code), type checks, tests, a build. A change merges only when they pass.
- Continuous delivery: every merged change is ready to deploy; a person clicks to release.
- Continuous deployment: every merged change deploys automatically, if all checks pass.
Branch protection makes it binding: the main branch only accepts pull requests whose required checks pass, and often a review too.
A GitHub Actions workflow
Workflows live in .github/workflows/ as YAML:
name: ci
on:
pull_request:
push:
branches: [main]
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12", cache: pip }
- run: pip install -r requirements-dev.txt
- run: ruff check . && mypy src
test:
needs: lint
runs-on: ubuntu-latest
strategy:
matrix:
python: ["3.11", "3.12"]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "${{ matrix.python }}", cache: pip }
- run: pip install -r requirements-dev.txt
- run: pytest -q
deploy:
needs: test
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@v4
- run: ./deploy.sh
env:
DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}
The parts: triggers (on), jobs that run on fresh machines, steps inside each job, needs for order, a matrix that runs a job once per Python version, caching of installed packages, and secrets injected only where needed. The production environment can require a person's approval.
Which job runs when?
Jobs declare what they need, and CI works out the order. Everything whose needs are met runs at the same time.
jobs = {"lint": [], "types": [], "test": ["lint", "types"], "build": ["test"], "deploy": ["build"]}
done, waves = set(), []
while len(done) < len(jobs):
ready = [j for j, needs in jobs.items() if j not in done and all(n in done for n in needs)]
waves.append(ready)
done |= set(ready)
for n, wave in enumerate(waves, 1):
print(f"wave {n}: {wave}")Lint and types run together, since neither needs the other. If either fails, the slow tests never start. That's the whole design: cheap checks first, and an expensive job only when it can pay off.
Show the answer
They're skipped, because they need lint (directly or through test)
needs builds a dependency graph. A failure skips everything downstream.
Designing a good pipeline
- Fast first. You check that a lamp is plugged in before you take it apart. Run quick checks (lint, types) before slow ones, so obvious mistakes fail in a minute.
- Parallel where possible. Jobs with no dependency between them run at the same time.
- Cache dependencies, so each run doesn't download the world.
- Build once, promote. Build one image, test it, then deploy that same image to staging and production. Rebuilding for production means shipping something you didn't test.
- Treat flaky tests as bugs. A smoke alarm that goes off at random soon gets ignored, and a test that fails at random teaches people to ignore red builds. Find the cause, often timing or shared state, and fix it.
- Keep secrets scoped. Give each job only the secrets it needs, and never print them.
For AI features, add an eval job when prompts or models change: run the eval set and fail if scores drop below a threshold. Keep it small enough to be cheap, and run the full suite before releases.
Why fast checks go first
Say most failures are caught by lint or types. Compare the wait before you hear about a broken change, with slow jobs first and fast jobs first.
jobs = [("lint", 1, 0.5), ("types", 2, 0.3), ("tests", 12, 0.2)] # name, minutes, share of failures caught def expected_wait(order): wait, spent = 0, 0 for name, minutes, share in order: spent += minutes wait += spent * share return wait print("fast first:", round(expected_wait(jobs), 1), "minutes") print("slow first:", round(expected_wait(jobs[::-1]), 1), "minutes")
Same jobs, same total time when everything passes, yet a broken change gets its answer about three times sooner. Those minutes add up across a team, and people stop context-switching while they wait.
Feature flags: deploy is not release
An electrician can wire a new lamp today, and you can flip the switch next week. A feature flag is that switch, in your code: if flags.enabled("new_checkout", user):. The code ships turned off, then you turn it on for staff, then 5% of users, then everyone. If something breaks, you turn it off in seconds, with no deploy.
In plain words: deploying puts the code on the servers. Releasing lets people use it. Flags let you do those at different times.
For percentage rollouts, each user must get a stable answer, like a fixed raffle number. Put the flag name and user id through a hash to get a bucket from 0 to 99, and turn the flag on when the bucket is below the percentage. Raising 5% to 20% keeps the first 5% in, and adds more. Then delete old flags: each one is a branch in your code that someone must understand.
Show the answer
Find and fix the cause, often timing or shared state
Flaky tests teach people to ignore failures. Fixing them keeps a red build meaningful.
Stable buckets
The same user always lands in the same bucket for a given flag.
import hashlib def bucket(flag, user_id): digest = hashlib.sha256(f"{flag}:{user_id}".encode()).hexdigest() return int(digest, 16) % 100 for user in ["ana", "bo", "chen", "dev"]: b = bucket("new_checkout", user) print(user, b, "on at 25%" if b < 25 else "off at 25%") print(bucket("new_checkout", "ana") == bucket("new_checkout", "ana"))
Including the flag name means each flag gets its own mix of users, so the same people aren't always first.
Each round, every job whose needs are all done joins the next stage. lint and docs run together; build waits for test.
bucket = 37 print([bucket < pct for pct in (10, 25, 50, 100)])
Show the answer
[False, False, True, True]
bucket < pct is False at 10 and 25, and True at 50 and 100, which gives [False, False, True, True]. They join once the rollout passes 37% and stay in as it grows, because the bucket never changes.
Build once
A tailor doesn't fit you in one suit and then deliver a different one. If you test one build and then rebuild for production, you've shipped something you never tested. Dependencies may have moved, and the compiler may have behaved differently. Build one image, give it a unique tag, and promote that same image from testing through staging to production.
Flaky tests are bugs
Retrying a failing test until it passes makes the pipeline green and teaches everyone to distrust it. A test that fails one time in twenty is telling you about a timing problem, either in the test or in your product. So quarantine it, file an issue with an owner, and fix or remove it soon. Re-running everything until it goes green hides real failures, and that's the worst of both worlds.
Rollouts need stable users
If you pick users at random on each request, someone may see the new checkout, then the old one, then the new one again. That's confusing for them, and it makes your data useless, since nobody had a consistent experience. Hashing the user id with the flag name, as this lesson's example does, gives each person the same answer every time. Their bucket stays the same as you widen the rollout, so people who already have the feature keep it.
Flags have a cost
Every flag adds a branch to your code, and old flags pile up. After a while, nobody knows what happens if an old one is turned off, and a stale flag has caused serious outages when someone flipped it years later. So when a rollout finishes, delete the flag and the old path. Give each flag an owner and an expected end date when you create it. Some teams add a test that fails if a flag is past its date, which keeps the cleanup from being forgotten.
Practice what you just read
- 3 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 5 review cards for spaced repetition
Try a quiz question
What's the difference between continuous delivery and continuous deployment?