Index Ai Academy
Phase 11 · Data engineering

Streams and windows

Events that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

A nightly batch has a comforting property: when it runs, yesterday is over. A stream never is. Clicks, payments and sensor readings arrive continuously, some of them minutes late, a few of them twice, and the features a live model needs, such as "purchases in the last five minutes", have to be computed while the data is still coming in. That raises a question a batch never has to answer: when is a count finished? This lesson builds the three ideas streaming systems use to answer it: two clocks, windows and watermarks.

What you'll be able to do

  • Tell a stream from a batch, and the time an event happened from the time it arrived
  • Group events into tumbling, sliding and session windows
  • Use a watermark to decide when a window is finished, and name what waiting longer buys and costs
  • Make a consumer give the right counts when the same event is delivered twice
Quick check
A guess before you start. A phone app records a tap at 10:00:50 while the phone is in a tunnel, and sends it when the signal returns at 10:02:15. A dashboard counts taps per minute. In which minute should this tap be counted?
  1. 10:00, the minute in which the tap happened
  2. 10:02, the minute in which the server received it
  3. Neither: a tap that arrives late should be discarded
  4. 10:01, halfway between the two times
Show the answer

10:00, the minute in which the tap happened

The question a dashboard answers is "how many taps happened in each minute", so the tap belongs to 10:00. Counting by arrival would move it to 10:02 and make the tunnel look like a quiet patch followed by a burst. The catch is that at 10:01, the server doesn't yet know the tap exists, which is the problem this lesson is about.

The idea: two clocks, windows and a watermark

This lesson stands on Reliability, where a retry sends the same request twice, and on counting by group from Describing data. First, a recall.

Quick check
From the lesson on rate limits, retries and idempotency: a payment request is retried because its reply was lost. What stops the customer from being charged twice?
  1. Sending the retry to a different server from the first attempt
  2. A longer timeout, so that the reply has more time to arrive
  3. Waiting a random time before the retry is sent
  4. A unique id sent with the request, which the server has already seen
Show the answer

A unique id sent with the request, which the server has already seen

The client sends the same idempotency key with the retry, and the server recognizes it and returns the first result without charging again. A stream consumer uses the same idea with event ids: an id it has seen is skipped.

Data that never ends

A batch is a finite set of data processed in one go: yesterday's orders. A stream is a sequence of events that keeps arriving, with no last row. Each event records one thing that happened: a click, a payment, a temperature reading.

Every event carries two times, and nearly every difficulty in stream processing comes from the gap between them.

  • Event time: when the thing happened, stamped by the device or service where it happened.
  • Processing time: when your system received it.

Most events arrive within a second or two. Some don't: a phone was offline, a server was retrying, a queue was backed up. Those arrive late, and so out of order.

# (event id, user, when it happened, when it arrived), in seconds
events = [
    (
        "e1", "ana", 5, 6
    ), ("e2", "ben", 20, 21), ("e3", "ana", 50, 135), ("e4", "cy", 62, 63),
    (
        "e5", "ben", 70, 71
    ), ("e6", "ana", 118, 119), ("e7", "cy", 58, 122), ("e8", "ben", 130, 131),
]

def per_minute(times):
    counts = {}
    for t in times:
        # the start of the minute this second falls in
        minute = t // 60 * 60
        counts[minute] = counts.get(minute, 0) + 1
    return dict(sorted(counts.items()))

print(
    "counted by arrival:   ",
    per_minute(arrived for eid, user, happened, arrived in events),
)
print(
    "counted by event time:",
    per_minute(happened for eid, user, happened, arrived in events),
)
It runs right here in your browser, with nothing to install.

Eight events, two different stories. By event time, the first minute was the busy one: four events. By arrival, it had two, and the third minute had three. Two events, e3 and e7, happened in the first minute and reached the server more than a minute later.

Counting by arrival is easy, since the server's own clock is always available, and it answers the wrong question: it measures the network, not the users. Counting by event time is what you want for anything you will chart, bill or train on. The price is that you can never be completely sure a minute is finished.

Quick check
What does this print?
# (id, happened, arrived)
events = [("a", 10, 11), ("b", 55, 56), ("c", 40, 130), ("d", 65, 66)]

late = [eid for eid, happened, arrived in events if arrived - happened > 60]
in_arrival_order = [
    eid for eid, happened, arrived in sorted(events, key=lambda e: e[2])
]
print(late, in_arrival_order)
  1. [] ['a', 'b', 'c', 'd']
  2. ['c'] ['a', 'c', 'b', 'd']
  3. ['c', 'd'] ['a', 'b', 'd', 'c']
  4. ['c'] ['a', 'b', 'd', 'c']
Show the answer

['c'] ['a', 'b', 'd', 'c']

Event c happened at second 40 and arrived at 130, 90 seconds later: the only one more than a minute late. Sorted by arrival, it comes last, after d, even though it happened before d. That is what out of order means.

Try it now

Step along the arrival clock. The two late events (marked) belong to minute 0 by when they happened.

Windows

You can't sum a stream, since it has no end. You can sum a slice of it. A window is a bounded stretch of the stream, and three shapes cover nearly every use.

Tumbling windows are fixed in length and don't overlap: each minute, each hour. Every event falls in exactly one. They answer "how many per minute".

Sliding windows are fixed in length and overlap: the last 60 seconds, recomputed every 30. An event falls in several. They answer "how many in the last minute, as of now", which is the shape of most live features.

Session windows have no fixed length. A session is a burst of one user's activity, and it ends when the user has been quiet for longer than a chosen gap. They answer "how long was the visit".

# event times of one stream, in seconds
times = [5, 20, 50, 58, 62, 70, 118, 130]

def tumbling(times, size):
    counts = {}
    for t in times:
        start = t // size * size
        counts[start] = counts.get(start, 0) + 1
    return dict(sorted(counts.items()))

def sliding(times, size, step):
    starts = range(0, max(times) + 1, step)
    return {s: sum(1 for t in times if s <= t < s + size) for s in starts}

def sessions(times, gap):
    out = []
    for t in sorted(times):
        if out and t - out[-1][1] <= gap:
            # close enough to the last event: same session
            out[-1] = (out[-1][0], t, out[-1][2] + 1)
        else:
            # too long a silence: a new session starts
            out.append((t, t, 1))
    return out

print("tumbling, 60 s:          ", tumbling(times, 60))
print("sliding, 60 s every 30 s:", sliding(times, 60, 30))
print("sessions, gap of 30 s:   ", sessions(times, 30))

Each window is named by the second it starts at. The tumbling windows hold 4, 3 and 1 events, which add up to 8: every event once. The sliding windows add up to 14, because each event is counted in two overlapping windows where two exist. The sessions are (start, end, events): one long visit from second 5 to 70, then, after a silence of 48 seconds, a short one.

The arithmetic for a tumbling window is worth knowing by heart: t // size * size rounds a time down to the start of its window.

Quick check
Tumbling windows are 15 seconds long. What does this print: the window each event falls in?
size = 15
print([t // size * size for t in (0, 14, 15, 44, 45, 61)])
  1. [15, 15, 30, 45, 60, 75]
  2. [0, 0, 1, 2, 3, 4]
  3. [0, 0, 15, 30, 45, 60]
  4. [0, 15, 15, 45, 45, 60]
Show the answer

[0, 0, 15, 30, 45, 60]

t // size is how many whole windows fit before t, and multiplying by the size gives that window's start. Second 14 is still in the window starting at 0. Second 15 starts the next. Second 44 is in the window from 30 to 44.

In your own words
In your own words: what are event time and processing time, and why can they differ?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: a watermark, windows and a set of ids seen

Everything below is plain Python on lists of events: the same logic a stream engine runs on many machines at once.

When is a window finished?

Now the hard part. A batch job counts yesterday after yesterday has ended. A stream has to publish the count for 10:00 while events from 10:00 may still be on their way. Wait too little and the count is wrong. Wait forever and there is no count.

Streaming systems settle this with a watermark: the system's running estimate of how far event time has progressed. The simplest watermark is

watermark=newest event time seen−allowed lateness\text{watermark} = \text{newest event time seen} - \text{allowed lateness}

In plain words: take the latest event time the system has seen so far, and subtract a waiting period you choose. The watermark says: "I believe everything that happened before this moment has arrived."

A window is closed, and its result published, once the watermark passes the window's end. An event that arrives for a window that has already closed is late. It is dropped, or set aside to correct the result afterward.

The allowed lateness is a dial, and here is what it trades. The stream below holds 2,000 events over ten minutes. Nine in ten arrive within 2 seconds, and the rest are delayed by 40 seconds on average.

import random

rng = random.Random(0)
stream = []
for i in range(2000):
    happened = rng.uniform(0, 600)
    delay = rng.uniform(0, 2) if rng.random() < 0.9 else rng.expovariate(1 / 40)
    stream.append((i, happened, happened + delay))
# the order in which they reach the server
stream.sort(key=lambda e: e[2])

def run(stream, size, lateness):
    counts, late, newest = {}, 0, 0.0
    for eid, happened, arrived in stream:
        start = int(happened // size * size)
        # how far the system believes event time has got
        watermark = newest - lateness
        if start + size <= watermark:
            # the watermark has passed this window's end: already published
            late += 1
        else:
            counts[start] = counts.get(start, 0) + 1
        newest = max(newest, happened)
    return counts, late

in_first_minute = sum(1 for eid, happened, arrived in stream if happened < 60)
for lateness in (0, 10, 30, 60, 120):
    counts, late = run(stream, 60, lateness)
    print(
        "wait", str(lateness).rjust(3), "s | too late to count:", str(late).rjust(3),
        "of 2000 |", round(100 * late / 2000, 1), "% | first minute counted:",
        counts[0], "of", in_first_minute,
    )

With no waiting, every window is published the moment a later event is seen, and 104 events, 5.2%, turn up too late to be counted. Waiting 30 seconds cuts that to 43. Waiting two minutes cuts it to 6, and every result is two minutes old before anyone sees it.

The last column shows the same trade on one window: the first minute really held 207 events, and the count published for it rises from 192 to 204 as the wait grows. It never quite reaches 207. There is always one more event that could arrive later still.

So completeness and latency pull against each other, and the right setting depends on the use. A live dashboard can publish early and be slightly low. A bill can't. When both are needed, publish an early figure and then correct it as late events arrive, which is what "set aside to correct afterward" means.

Quick check
A stream uses tumbling windows of 60 seconds and an allowed lateness of 20 seconds. The newest event time seen so far is 135. What does this print: the watermark, and which of the windows starting at 0, 60 and 120 are closed?
size, lateness, newest = 60, 20, 135
watermark = newest - lateness
print(watermark, [start for start in (0, 60, 120) if start + size <= watermark])
  1. 155 [0, 60]
  2. 115 [0, 60]
  3. 115 [0]
  4. 135 [0, 60]
Show the answer

115 [0]

The watermark is 135 − 20 = 115. A window is closed when the watermark has passed its end. The first window ends at 60, which is before 115: closed. The second ends at 120, which the watermark hasn't reached: still open, waiting for latecomers.

Quick check
A team raises a stream's allowed lateness from 10 seconds to 5 minutes. What changes?
  1. Fewer events are dropped as late, and every result is published later
  2. Fewer events are dropped as late, and results are published sooner
  3. Nothing: allowed lateness affects only how events are stored
  4. More events are dropped as late, and results are published later
Show the answer

Fewer events are dropped as late, and every result is published later

A longer wait keeps each window open for longer, so more stragglers make it in. It also holds back every window's result by the same amount. Completeness is bought with latency.

The same event, twice

One more fact of streams. When a sender isn't sure an event was received, it sends it again. Sending again is the only safe choice, since the alternative is losing data, and it means a consumer must expect duplicates. This is called at-least-once delivery: every event arrives, some more than once.

The fix is the one from the reliability lesson. Give every event a unique id where it is created, and make processing idempotent: handling the same event twice leaves the same result as handling it once.

# (event id, event time)
deliveries = [("e1", 5), ("e2", 20), ("e2", 20), ("e3", 50), ("e1", 5), ("e4", 62)]

naive = {}
for eid, t in deliveries:
    naive[t // 60 * 60] = naive.get(t // 60 * 60, 0) + 1

seen, careful = set(), {}
for eid, t in deliveries:
    if eid in seen:
        continue                                   # this event has been counted already
    seen.add(eid)
    careful[t // 60 * 60] = careful.get(t // 60 * 60, 0) + 1

print("counting every delivery:", naive)
print("counting each event once:", careful)

Four events were delivered six times. Counted naively, the first minute shows 5 events where there were 3. With a set of ids already seen, each event counts once however many times it is delivered.

In a real system that set can't grow forever, so it holds only the ids from windows still open: once a window closes, its ids can be forgotten, since anything arriving for it would be late anyway. The watermark does double duty.

Go deeper Where this runs

In production the stream itself usually lives in a log such as Kafka: an append-only sequence of events that many consumers read at their own pace, each remembering how far it has got. The windowing, watermarks and state shown here are what engines such as Flink and Spark Structured Streaming provide, spread across many machines and able to survive one of them failing. The ideas don't change. What a stream processor adds is doing them at scale without losing its place.

Drill

Count each event once, in the window where it happened

Write count_once(deliveries, size). deliveries is a list of (event_id, event_time) in the order they arrived, possibly with repeats and out of order. Return a tuple: a dictionary from each tumbling window's start to the number of distinct events in it, sorted by window start, and the number of deliveries that were skipped as duplicates.

def count_once(deliveries, size):
    return {}, 0


sample = [("e1", 5), ("e2", 20), ("e2", 20), ("e3", 50), ("e1", 5), ("e4", 62)]
print(count_once(sample, 60))
In the lesson, your code is checked as you write it.
Drill

Split a user's events into sessions

Write sessions(times, gap). times are the event times of one user, in any order. Sort them, then group them: an event joins the current session if it comes at most gap seconds after the session's latest event, and otherwise starts a new one. Return a list of (start, end, count), one tuple per session, in time order.

def sessions(times, gap):
    return []


print(sessions([5, 20, 50, 58, 62, 70, 118, 130], 30))
In the lesson, your code is checked as you write it.
Worked example

Worked example: a live feature, computed as events arrive

Put event time, a sliding window and duplicate handling together: for each payment, compute "payments by this card in the 60 seconds up to and including this one", the kind of feature a fraud model reads.

# (payment id, card, event time in seconds), in arrival order
payments = [
    (
        "p1", "card-7", 3
    ), ("p2", "card-7", 15), ("p3", "card-2", 20), ("p2", "card-7", 15),
    (
        "p4", "card-7", 40
    ), ("p5", "card-7", 58), ("p6", "card-2", 95), ("p7", "card-7", 110),
]

history, seen = {}, set()          # card -> event times of its payments so far
for pid, card, t in payments:
    if pid in seen:
        print(pid, card, "at", t, "| duplicate delivery, skipped")
        continue
    seen.add(pid)
    times = history.setdefault(card, [])
    times.append(t)
    recent = sum(1 for other in times if t - 60 < other <= t)
    print(
        pid, card, "at", t, "| payments by this card in the last 60 s:", recent,
        "<- unusual" if recent >= 4 else "",
    )
In the lesson you can change this and run it.

Each payment is judged with the window that ends at its own event time, so the feature is point-in-time correct in the sense of the feature pipelines lesson: it counts what had happened by then and nothing after. The second delivery of p2 is recognized by its id and skipped, so card-7's count isn't inflated by a retry. By its fourth distinct payment within a minute, at second 58, the card is flagged. At second 110 the payments from seconds 3, 15 and 40 have slid out of the window and the count is back down. This small loop is what a streaming feature pipeline does, at many thousands of events a second.

A plot makes the two clocks visible side by side:

import matplotlib.pyplot as plt

events = [
    (5, 6), (20, 21), (50, 135), (62, 63), (70, 71), (118, 119), (58, 122), (130, 131),
]
by_event, by_arrival = [0, 0, 0], [0, 0, 0]
for happened, arrived in events:
    by_event[happened // 60] += 1
    by_arrival[arrived // 60] += 1
xs = [0, 1, 2]
plt.bar([x - 0.2 for x in xs], by_event, width=0.4, label="by event time")
plt.bar([x + 0.2 for x in xs], by_arrival, width=0.4, label="by arrival time")
plt.xticks(xs, ["0 to 59 s", "60 to 119 s", "120 to 179 s"])
plt.ylabel("events counted")
plt.legend()
plt.show()
print(by_event, by_arrival)

The pairs of bars differ, and only event time answers "how many happened". Arrival time measures the network.

Quick check
Look at the plot. Which minute holds 4 events by event time but only 2 by arrival time?
  1. The first minute
  2. The third minute
  3. None: the bars match
  4. The second minute
Show the answer

The first minute

The first minute holds 4 events by event time and 2 by arrival, because e3 and e7 arrived over a minute late.

In your own words
A colleague wants per-minute counts from a stream to be both instant and exactly right. What do you tell them?

Write a sentence or two, then compare it with a good answer in the lesson.

In a real system the events sit in a log, and an engine keeps the windows, the watermark and the ids already seen. Your job is to choose the settings and prove the numbers.

Where it comes up in a week

A fraud model reads "payments by this card in the last 10 minutes". You set the allowed lateness with the team that owns the number: a dashboard can be a few seconds low, a bill cannot. When the stream and the nightly batch disagree, check which clock each one used first. Two failures to look for: a consumer that commits its offset before it has handled the event (a crash then loses it), and a consumer that handles an event twice because it never skips an id it has seen.

Goes deeper in

In your own words
A fraud feature is "payments by this card in the last 10 minutes". Which kind of window is it, and why wouldn't a tumbling window do?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 6 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 12 review cards for spaced repetition

Try a quiz question

Why do stream processors group events by event time and not by the time they arrive?

  1. Arrival time reflects network delays, not when things happened
  2. Arrival time isn't recorded anywhere in a stream
  3. Event time is always later than arrival time
  4. Grouping by arrival time needs more memory

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 11 · about 34 hours of study

Professional practice

Design, ship and run real systems, work well in a team, and get hired 43 lessons 1 free
  1. System design fundamentals FreeRequirements, rough numbers, the standard parts and where systems break.40 min
  2. Caches, queues and background jobsSpeed reads with caches, and move slow work to queues that retry safely.40 min
  3. Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
  4. Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
  5. CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
  6. Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
  7. Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
  8. Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
  9. Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
  10. Reliability and incidentsSLOs, error budgets, alerts, on-call and blameless postmortems.40 min
  11. Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
  12. Measuring an AI feature from its logsTyped views over event logs, metric specs, cost from a price table, and rollups you can rerun.40 min
  13. A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
  14. Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
  15. Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
  16. Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
  17. Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
  18. Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
  19. Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
  20. Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
  21. Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
  22. A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
  23. Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
  24. Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
  25. Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
  26. Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
  27. Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
  28. Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
  29. Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
  30. Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
  31. Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
  32. A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
  33. What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
  34. Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
  35. Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
  36. Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
  37. Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
  38. Interviews and your portfolioShow your work, and solve interview problems the way professionals do.40 min
  39. Code reviewWhat to look for, how to give and take feedback, and automating the routine checks.35 min
  40. Reading unfamiliar codeFind your way around a big codebase fast, with tools that map it for you.35 min
  41. Working with AI coding toolsGive agents good context, keep tasks small, and check every change they make.35 min
  42. Capstone: ship a production AI serviceAn AI-backed service with validation, caching, rate limits, fallbacks, logging and evals.45 min
  43. Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write