Regular expressions

A pattern for text

A regular expression, or regex, is a small pattern language for text. Most characters match themselves; a few special ones describe kinds of text. \d means "any digit", and + means "one or more of the thing before". So \d+ matches a run of digits:

import re

line = "Order 50231: 2 tents, 1 stove, shipped in 3 days"
print(re.search(r"\d+", line).group())
print(re.findall(r"\d+", line))

re.search finds the first match and returns a match object. Its .group() is the text that matched. re.findall returns every match as a list of strings.

The r before each pattern makes it a raw string: Python leaves backslashes alone, so r"\d" reaches the re module as backslash-d. Without the r, some backslash pairs mean something else to Python first ("\b" is a backspace character). Always write patterns as raw strings.

The building blocks

To match a special character itself, put a backslash before it: \. is a real dot and \$ a real dollar sign. The practice shop's product codes, like TE-101, are two capital letters, a hyphen and three digits: [A-Z]{2}-\d{3}.

import re

note = "Swap TE-101 for TE-104, and restock AP-122. Ignore te-999 and ABC-1234."
print(re.findall(r"[A-Z]{2}-\d{3}", note))
print(re.findall(r"\b[A-Z]{2}-\d{3}\b", note))

In plain words: read a pattern left to right, one piece at a time. [A-Z]{2}-\d{3} says "a capital letter, twice; then a hyphen; then a digit, three times". \b adds "and nothing word-like touching it".

The first pattern also finds "BC-123" hiding inside "ABC-1234". \b is a word boundary: the edge between a word character and anything else. With \b at both ends, the code must stand on its own, so the inside match is gone.

import re
print(re.findall(r"\d{2,3}", "1 22 333 4444"))

Groups: pulling out the pieces

Round brackets make a group: a part of the match you can read separately. Give a group a name with (?P<name>...) and the match hands you a dictionary:

import re

line = "2025-03-14 09:12 ORDER #50231 store=portland total=$200.00"
m = re.search(r"#(?P<order>\d+) store=(?P<store>\w+) total=\$(?P<total>[\d.]+)", line)
print(m.group("order"), m.group("store"))
print(m.groupdict())

for found in re.finditer(r"(\w+)=(\S+)", line):
    print(found.group(1), "->", found.group(2))

• m.group("order") reads one named group; m.group(1) reads the first group by number.
• m.groupdict() gives every named group at once, ready to become a record.
• re.finditer gives a match object for every match, so a loop can read each one's groups.

Watch out: when a pattern has groups, re.findall returns the groups, not the whole match. With one group you get a list of strings, and with several, a list of tuples.

Everything a regex captures is text. "50231" is a string until you call int() on it.

import re

line = "sync store=seattle mode=full retries=3 dry_run=no"
settings = {}

Replace and split

re.sub(pattern, replacement, text) replaces every match. In the replacement, \1 or \g<name> puts a group back in. re.split splits on a pattern instead of one fixed character:

import re

email = "Hi, my order #50231 hasn't arrived. Card ending 4242. Call me on 503-555-0134."
print(re.sub(r"\d{3}-\d{3}-(\d{4})", r"***-***-\1", email))
print(re.sub(r"\s+", " ", "too    many\n  spaces"))

tags = "hiking; camping ,budget;  winter"
print(re.split(r"\s*[;,]\s*", tags))

The phone number keeps only its last four digits: the group captures them and \1 puts them back. \s+ collapses any run of spaces and newlines into one space, a classic cleanup. The split pattern means "a semicolon or comma, with any spaces around it", which handles the messy separators in one go.

Whole-string checks, and readable patterns

re.search is happy if the pattern appears anywhere. To check that a whole string has a shape, like a product code a customer typed, use re.fullmatch, which must match from the first character to the last. (^ and $ anchor a pattern to the start and end too, but fullmatch says it more plainly.)

import re

for code in ["TE-101", "TE-101x", " TE-101", "te-101"]:
    print(repr(code), bool(re.fullmatch(r"[A-Z]{2}-\d{3}", code)))

Long patterns get hard to read. re.VERBOSE lets you spread one over several lines, with spaces ignored and # comments allowed:

PHONE = re.compile(r"""
    \(? (\d{3}) \)?     # area code, maybe in brackets
    [\s.-]?             # an optional separator
    (\d{3}) [\s.-]? (\d{4})
""", re.VERBOSE)

re.compile turns a pattern into an object with the same methods (PHONE.search(text)). It's handy for a pattern you use in many places, and it keeps the pattern's name next to its meaning.

When not to use a regex

Regular expressions are great for small, flat patterns in text. They're the wrong tool when the text has a structure of its own:

• JSON: use json.loads, which handles nesting, escapes and quotes correctly.
• CSV: use the csv module or pandas. A regex splitting on commas breaks on "Pine & Peak, LLC".
• HTML: use an HTML parser such as Beautiful Soup. Tags nest, attributes come in any order, and pages are often slightly broken.
• Email addresses: the full rules are famously complicated. A simple check for "something @ something . something" catches typos; to really know an address works, send it a message.

A good rule: if you're writing a regex longer than a line or two to parse something, look for a parser first. Keep regexes for finding, checking and cleaning.

import re

def mask(text):
    pass

print(mask("Orders #50231 and #7 shipped; #120 is late."))

import re

LOG = """2025-03-14 09:12:05 ORDER #50231 store=portland sku=TE-101 qty=1 total=$200.00
2025-03-14 09:13:40 heartbeat ok
2025-03-14 09:20:11 ORDER #50232 store=online sku=FO-120 qty=3 total=$60.00
2025-03-14 09:31:52 ORDER #50233 store=denver sku=PA-109 qty=1 total=$1,095.00"""

LINE = re.compile(r"""
    (?P<date>\d{4}-\d{2}-\d{2}) \s (?P<time>\d{2}:\d{2}:\d{2}) \s ORDER \s
    \#(?P<order>\d+) \s store=(?P<store>\w+) \s sku=(?P<sku>[A-Z]{2}-\d{3}) \s
    qty=(?P<qty>\d+) \s total=\$(?P<total>[\d,]+\.\d{2})
""", re.VERBOSE)

records = []
for line in LOG.splitlines():
    m = LINE.fullmatch(line)
    if not m:
        continue
    row = m.groupdict()
    row["order"] = int(row["order"])
    row["qty"] = int(row["qty"])
    row["total"] = float(row["total"].replace(",", ""))
    records.append(row)

print(len(records), "orders")
print(records[2])
print(sum(r["total"] for r in records))