Shell scripts that fail safely

Giving a script arguments

Scripts take arguments the way commands do. Inside the script, $1 is the first argument, $2 the second and $# how many there are, so bash scripts/report.sh logs/old.log runs with $1 set to logs/old.log. Under set -u a missing argument is an error, so give a default when there's a sensible one: LOG="${1:-logs/access.log}" uses the first argument if there is one and the default if not.

When there's no sensible default, check and explain how to call the script:

if [ $# -lt 1 ]; then
  echo "usage: deploy.sh ENVIRONMENT" >&2
  exit 2
fi

>&2 sends the message to standard error, the stream meant for problems, so it doesn't get mixed into the real output when someone pipes the script into something else. A usage line like this is the cheapest documentation a script can have, and it's the first thing a teammate sees when they run it wrong.

set -euo pipefail

Put this on the line after the shebang in every script you write:

• -e: stop as soon as a command fails, instead of carrying on;
• -u: treat an unset variable as an error, so a typo like $LGO stops the script instead of quietly becoming an empty string (imagine rm -rf "$BUILD_DIR/" with BUILD_DIR unset);
• -o pipefail: a pipeline fails if any stage fails, not only the last one. Without it, curl -sf URL | jq . succeeds even when curl failed, because jq, the last stage, didn't.

The three together turn a script that limps on from a failure into one that stops at the line where it went wrong, which is the line you'll read first when you debug it.

curl -sf https://api.trailhead.test/products | wc -l

A gotcha: grep and set -e

One gotcha follows from these rules. grep exits with a failure code when it finds nothing, so under set -e a search with no matches stops the script. With pipefail the same goes for a pipeline such as grep ' 500 ' "$LOG" | wc -l, even though wc -l printed 0. When "no matches" is a normal answer, say so explicitly with || true: grep -c ' 500 ' "$LOG" || true prints the count, 0 included, and the line succeeds. || runs the next command only when the one before it failed, the mirror image of &&.

Exit codes

Every command finishes with an exit code: 0 means success, anything else means failure, and the number often says which kind. grep uses 1 for "no match" and 2 for real errors; curl -f uses 22 for an HTTP error. && runs the next command only after a 0 and || only after a failure; a script's own exit code is that of its last command, or whatever it passes to exit. That's what CI systems read: a script that exits non-zero turns the job red.

So end a check script with exit 1 when something's wrong, after printing what it was. A script that prints "ERROR" but exits 0 looks fine to every machine that runs it.

When a script should become Python

Shell scripts are the right tool for gluing commands together: run the tests, build the image, copy a file, call a deploy command. They get fragile once they need data structures, arithmetic, error handling beyond "stop", or careful handling of untrusted input, where quoting mistakes become security holes. A rough rule: past about fifty lines, or the first time you need an array of records, write it in Python, which you can also test.

Whichever you choose, keep scripts in the repository next to the code they serve, run them in CI, and give each one a short comment at the top saying what it does and how to run it. A script nobody can read is a script nobody dares to change.