Text, bytes and encodings

Always name the encoding

open(path) in text mode decodes for you, using a default encoding that depends on the operating system. On many Windows computers it isn't UTF-8, so a file that reads fine on a Mac can crash on Windows. Name it every time:

with open("menu.txt", "w", encoding="utf-8") as f:
    f.write("Crème brûlée\nPão de queijo\n")

with open("menu.txt", encoding="utf-8") as f:
    print(f.read().splitlines())

with open("menu.txt", "rb") as f:
    raw = f.read()
print(len(raw), "bytes on disk")

try:
    raw.decode("ascii")
except UnicodeDecodeError as e:
    print("ascii can't read it:", e.reason, "at byte", e.start)
print(raw.decode("ascii", errors="replace").splitlines()[0])

errors= chooses what happens with bytes that don't fit: "strict" (the default) raises UnicodeDecodeError; "replace" puts in the replacement character �; "ignore" drops them silently. Prefer strict, so problems are loud. Use "replace" only when a few damaged characters are acceptable, as in a log viewer. Never use "ignore" on data you care about: it deletes letters from people's names without a trace.

Opening in binary mode ("rb") gives you the bytes untouched, which is right for images, zip files and anything that isn't text.

HTTP responses work the same way. With the requests library, response.content is the raw bytes and response.text is those bytes decoded, using the encoding the server announced in its headers, or a guess when it didn't say. If .text comes out garbled, check response.encoding; setting it to "utf-8" before reading .text often fixes the problem. Databases and spreadsheet exports have encodings too, and the same rule applies: find out what the source used, and decode it on purpose rather than by accident.

s = "naïve café"
print(len(s), len(s.encode("utf-8")))

Mojibake: decoded with the wrong encoding

When bytes are decoded with the wrong encoding, you get mojibake: garbled text like "Café". It's not random. UTF-8 encodes "é" as two bytes, and an older Windows encoding, cp1252, reads those two bytes as the two characters "é". Because the mistake is systematic, it can be reversed:

original = "Café Lucía"
garbled = original.encode("utf-8").decode("cp1252")
print(garbled)
fixed = garbled.encode("cp1252").decode("utf-8")
print(fixed, fixed == original)

def looks_garbled(text):
    return any(marker in text for marker in ("Ã", "Â", "â€"))

print(looks_garbled(garbled), looks_garbled(original))

Reversing it means encoding back to the bytes with the wrong encoding, then decoding them correctly. Only do this when you're confident it's mojibake: the telltale "Ã" and "â€" patterns are a good sign. For messy real data, the third-party ftfy library ("fixes text for you") recognizes many more patterns.

A related trap is the byte-order mark (BOM): three invisible bytes, \\xef\\xbb\\xbf, that some programs, notably Excel, put at the start of a UTF-8 file. Decoded as plain UTF-8, it becomes an invisible character glued to your first column name, so row["sku"] mysteriously fails. The encoding "utf-8-sig" reads and drops it.

def to_text(raw):
    return raw.decode("utf-8")


print(to_text("Café".encode("utf-8")), to_text("Café".encode("cp1252")))

When equal-looking text isn't equal

Unicode can write some characters more than one way. "é" can be one character (é, U+00E9), or an "e" followed by a combining accent (U+0301). They look identical but compare unequal, so a search or a duplicate check misses them. Normalization picks one form:

import unicodedata

a = "Lucía"
b = "Luci\u0301a"
print(a, b, a == b, len(a), len(b))
nfc = [unicodedata.normalize("NFC", s) for s in (a, b)]
print(nfc[0] == nfc[1])

print("STRASSE".lower() == "straße".lower(), "STRASSE".casefold() == "straße".casefold())

def match_key(name):
    return unicodedata.normalize("NFC", name).casefold().strip()

print(match_key("  Lucía ") == match_key("LUCI\u0301A"))

NFC (the composed form) is the usual choice for storing text. casefold() is a stronger lower() meant for comparing: it also folds characters like "ß" to "ss". A "match key" that normalizes, casefolds and strips is the standard way to compare names, emails or product titles typed by people.

Bytes inside text: base64

Some formats can only carry text: JSON, email bodies, URLs, many APIs. To send bytes through them, such as an image for a vision model or a file attachment, encode the bytes as base64. It writes any bytes using 64 safe characters (letters, digits, + and /):

import base64
import json

icon = bytes([137, 80, 78, 71, 13, 10, 26, 10])
text = base64.b64encode(icon).decode("ascii")
payload = json.dumps({"filename": "icon.png", "data": text})
print(payload)
back = base64.b64decode(json.loads(payload)["data"])
print(back == icon, len(icon), "bytes became", len(text), "characters")

base64 makes data about a third larger, and it is not encryption: anyone can decode it. You've already met it in this course: the academy's charts travel from Python to the page as base64 text, and the Messages API accepts images the same way.

import unicodedata

lines = [
    "\ufeffDrizzle rain jacket".encode("utf-8"),
    "Crème stove fuel".encode("cp1252"),
    "Café mug".encode("utf-8"),
    "Café mug".encode("utf-8"),
    "Luci\u0301a's trail mix".encode("utf-8"),
]

def to_text(raw):
    try:
        return raw.decode("utf-8-sig")
    except UnicodeDecodeError:
        return raw.decode("cp1252")

def repair(text):
    if any(m in text for m in ("Ã", "Â", "â€")):
        try:
            return text.encode("cp1252").decode("utf-8")
        except UnicodeError:
            return text
    return text

clean = [unicodedata.normalize("NFC", repair(to_text(raw))) for raw in lines]
for name in clean:
    print(repr(name))
print(len(set(clean)), "distinct names from", len(clean), "lines")