Binary files

Bigger numbers take several bytes

One byte only counts to 255, so larger numbers are spread over two, four or eight bytes. The catch is the order. Write 258 in base 256 and it's 1 × 256 + 2, so two bytes, 1 and 2. Big-endian stores the big part first, 01 02, the way we write digits. Little-endian stores the small part first, 02 01, which is what most processors do in memory. Neither is wrong; a file format simply has to say which it uses, and you have to read it the same way.

data = bytes([1, 2])
print(int.from_bytes(data, "big"))        # 1 * 256 + 2
print(int.from_bytes(data, "little"))     # 2 * 256 + 1

steps = 70_000
print(steps.to_bytes(4, "little").hex(" "), steps.to_bytes(4, "big").hex(" "))
print(int.from_bytes(steps.to_bytes(4, "little"), "little"))   # round trip

In plain words: endianness is which end of the number comes first. Read with the wrong one and you get a different, usually absurd, number: a sensor that "counted" 513 hikers when it meant 258.

print(int.from_bytes(bytes.fromhex("2c 01"), "little"))

Fixed-size records with struct

Devices and many file formats write records: the same few fields, in the same order, over and over. The trail club's counters do exactly this. Each record is 8 bytes: a 2-byte station number, a 4-byte timestamp (seconds since 1970), and a 2-byte hiker count, all little-endian. The struct module packs and unpacks such records from a short format string: < for little-endian (> for big), H for a 2-byte unsigned number, I for a 4-byte one, f for a 4-byte floating-point number. The < or > prefix matters for size too: without it, struct pads fields the way the machine's memory does, and struct.calcsize("HIH") is 10, not 8.

import struct

RECORD = "<HIH"                        # station, timestamp, count
print(struct.calcsize(RECORD), "bytes per record")

raw = struct.pack(RECORD, 3, 1_750_000_000, 42) + struct.pack(RECORD, 7, 1_750_003_600, 15)
print(raw.hex(" "))

for station, ts, count in struct.iter_unpack(RECORD, raw):
    print(station, ts, count)

calcsize confirms the record is 8 bytes, pack turns values into bytes, and iter_unpack walks a run of records for you, one tuple each. If the data's length isn't a whole number of records, iter_unpack raises an error, which is usually a sign the file was cut off or the format string is wrong. Other codes you'll meet: B for one unsigned byte, h and i for signed 2- and 4-byte numbers (which can be negative), and d for an 8-byte floating-point number.

import struct

RECORD = "<HIH"


def read_counts(raw):
    station, ts, count = struct.unpack(RECORD, raw[:8])
    return {station: count}


raw = struct.pack(RECORD, 3, 1_750_000_000, 42) + struct.pack(RECORD, 7, 1_750_003_600, 15) + struct.pack(RECORD, 3, 1_750_007_200, 8)
print(read_counts(raw))

A file's real type is in its first bytes

A file name is a label anyone can change. The bytes at the start of most formats, the signature (or "magic number"), are what programs actually trust:

That last row is a nice surprise: a Word or Excel file is a zip archive of XML files, so it starts like any zip.

For uploads, this is the check that matters. A user can rename invoice.exe to photo.jpg, and the extension check passes. Reading the first few bytes tells you what the file really is. Combine it with a size limit before reading anything large, and treat a passing signature as "probably that format", not "safe": the rest of the file can still be broken or hostile, so it should go to a well-maintained library to open.

def sniff(data):
    return None


def allowed_image(name, data):
    return name.lower().endswith((".png", ".jpg", ".jpeg"))


print(sniff(b"\x89PNG\r\n\x1a\n...."), allowed_image("photo.jpg", b"MZ\x90\x00"))

import struct

def png_header(width, height):
    ihdr = struct.pack(">II", width, height) + bytes([8, 2, 0, 0, 0])   # size, then 5 settings bytes
    return b"\x89PNG\r\n\x1a\n" + struct.pack(">I", len(ihdr)) + b"IHDR" + ihdr

def png_size(data):
    if not data.startswith(b"\x89PNG\r\n\x1a\n") or data[12:16] != b"IHDR":
        raise ValueError("not a PNG")
    return struct.unpack(">II", data[16:24])

for w, h in [(1200, 800), (30_000, 30_000)]:
    data = png_header(w, h)
    width, height = png_size(data)
    print(width, height, "ok" if width * height <= 40_000_000 else "too big: refuse before decoding")