Pipes and text tools in the shell
grep, wc, sort, uniq, head and cut, chained with pipes to answer questions about logs and data.
Create a free account Try part of it here
Why it matters
When a server misbehaves at 2am, the evidence is usually in text: logs, CSV exports, config files. Engineers answer most questions about that text without writing a program, by chaining small shell tools with pipes: how many checkouts failed, which page is slowest, who ordered most. It's also how you look around a machine you've just logged into, where your editor and your Python environment may not be. This lesson builds the habit one tool at a time, in a simulated terminal with a small shop's logs.
What you'll be able to do
- Search files and streams with grep, and count with wc
- Chain small tools with pipes to answer a question in one line
- Sort, count and rank with sort, uniq and head
- Pull fields out of logs and CSV files with cut
What it covers
- Small tools, joined by pipes
- grep: keep the lines that match
- Sort, count, rank
- When to stop and write Python
Show the answer
One shell command that counts the matching lines
grep finds the matching lines and its -c flag counts them, in one line and in seconds, without loading the whole file into an editor.
Small tools, joined by pipes
Each classic shell tool does one job on lines of text: grep keeps the lines that match, wc counts them, sort puts them in order, head keeps the first few. The pipe, |, sends one command's output straight into the next command as its input, so small tools combine into answers. Think of an assembly line: each station does one thing to whatever arrives and passes it on, and you can add, remove or reorder stations freely:
grep POST logs/access.log | grep 500 | wc -l
Read it left to right: take the POST requests, keep the ones with 500 in them, count the lines. Nothing is written to disk in between. Each tool reads lines in and writes lines out, which is why any of them can sit anywhere in a pipeline.
This idea comes from Unix, the system macOS and Linux descend from, and it has lasted fifty years because it scales in both directions. The same pipeline works on ten lines or ten million, and each tool is small enough to learn in a minute. The tools here work the same way on the Mac terminal, on a Linux server you've connected to with SSH, and inside most Docker containers, so the habits travel with you.
Show the answer
The POST lines that also contain 500
Each pipe passes the previous command's output along: first the POST lines, then only those with 500, which wc -l counts.
grep: keep the lines that match
grep PATTERN FILE prints every line containing the pattern. Four flags cover most needs:
-iignores case, sogrep -i erroralso finds ERROR;-vinverts the match: lines that don't contain it;-cprints how many lines matched instead of the lines;-nputs each line's number in front of it.
The pattern is a regular expression, so ' 500 ' with spaces matches the status code and not a duration like 1500ms. Quote patterns that contain spaces or special characters. And wc -l counts lines, whether they come from a file or a pipe.
Regular expressions are a language of their own, but a few pieces go a long way. A dot matches any one character, ^ anchors a match to the start of the line and $ to its end, and a|b matches either side. So grep '^2026-11-03 09:1' finds the requests in the ten minutes after nine, and grep -E 'checkout|cart' finds both kinds of request. When a pattern behaves oddly, run grep on its own and look at what it matched before adding it to a longer pipeline.
A simulated terminal with a shop's logs. Nothing here touches your real files. Each task is checked by what your command prints.
Write a sentence or two, then compare it with a good answer in the lesson.
Sort, count, rank
sort orders lines alphabetically; sort -n orders them as numbers and -r reverses. uniq collapses neighboring repeated lines, and uniq -c puts a count in front of each. Because uniq only looks at neighbors, you sort first so equal lines end up together. Then sort -rn ranks the counts, and head -n 3 keeps the top three. This five-step shape answers "what's most common?" for almost any text:
cut -d ' ' -f 5 logs/access.log | sort | uniq -c | sort -rn | head -n 3
cut picks out fields: -d ' ' says fields are separated by spaces and -f 5 takes the fifth, the status code here. For a CSV file the separator is a comma: cut -d , -f 2.
Two more tools round out the kit. head -n 5 keeps the first five lines and tail -n 5 the last five, handy for peeking at a big file or keeping the top of a ranking. And sort -u sorts and drops duplicates in one step, the quickest way to list the distinct values in a column, such as every page a log mentions or every customer in an export. With these six tools you can answer most first questions about a log before anyone has opened a dashboard.
sort notes.txt | uniq -c
Show the answer
1 a, then 2 b
sort gives a, b, b; uniq -c counts each run: 1 a and 2 b, in sorted order. Without the sort you'd get 1 b, 1 a, 1 b.
Show the answer
uniq only merges repeated lines that sit next to each other
uniq compares each line with the one before it. Sorting brings equal lines together, so each value gets a single count.
Show the answer
> replaces the file's contents; >> adds to the end
One arrow overwrites, two arrows append. That's true at the end of a pipeline too: grep ' 500 ' logs/access.log > errors.txt saves just the matches.
When to stop and write Python
Pipelines are perfect for questions you ask once: what failed, how many, which is most common. They get hard to read once they need conditions, arithmetic across fields or state that carries between lines, and they quietly go wrong on messy data, like a CSV field that contains a comma inside quotes. That's the moment to switch to Python with the csv module, or to load the data into a database. A useful rule of thumb: if a pipeline no longer fits on one line, or you'd need to explain it to a teammate in more than a sentence, it has become a program, and programs deserve a real language, tests and version control.
A good habit either way: build a pipeline one stage at a time, looking at the output after each | before adding the next, exactly as you would test small functions.
Pipelines also earn a place in automation. A CI job can fail the build when grep -c TODO finds too many, a deploy script can check curl output before switching traffic, and a cron job can mail you the five slowest requests every morning. The next two lessons build on this one: reading JSON from APIs with curl and jq, and turning commands into scripts that stop safely when something goes wrong.
Show the answer
Totaling refunds per customer from a CSV with quoted commas
Quoted commas break cut's simple splitting, and summing by customer needs arithmetic and state. Python's csv module handles both correctly.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- a practice exercise that checks your code
- a 6-question quiz
- 5 review cards for spaced repetition
Try a quiz question
What does grep -v GET access.log print?