HTTP from the wire up

Four steps under one call

Picture posting a letter to a company you've never written to. First you look up its address. Then you need a route that gets the letter there safely. You seal it in an envelope, so nobody can read it on the way. Only then does the letter itself matter.

A call like requests.get("https://api.gradient.test/v1/campaigns") goes through the same four steps, in order:

1. DNS is the address book. It turns the name api.gradient.test into the server's numeric address, called its IP address.
2. TCP opens a connection to that address. A server has numbered doors called ports: 443 for HTTPS and 80 for plain HTTP. The internet only promises to try to deliver small packets of data, and some get lost or arrive out of order. TCP fixes that, so your program sees a steady stream of bytes, in the right order. Think of a phone line that never drops a word.
3. TLS seals the envelope. It encrypts the stream, so nobody in between can read it. It also checks the server's certificate, which is proof that the server is who it claims to be; so TLS adds encryption and proof of identity. The certificate proves this really is api.gradient.test. HTTPS is HTTP inside TLS.
4. HTTP is the letter itself: a request in a fixed text format, and a reply in the same style.

Each step only relies on the one below it. HTTP doesn't care how TCP delivers its bytes, which is why the same HTTP works over Wi-Fi, fiber or a phone network.

In plain words: find the address, open a reliable line, seal it, then send a letter in an agreed format.

A request is text

The version called HTTP/1.1 sends everything as readable text, so it's the best one to learn from. Here's a complete request, exactly as it travels:

POST /v1/campaigns?dry_run=1 HTTP/1.1
Host: api.gradient.test
Content-Type: application/json
Content-Length: 33

{"name": "Spring", "budget": 500}

Read it from the top, like a form:

• The request line says what you want: the method (POST means "here's something new"), the path (with any query after the ?), and the protocol version.
• The headers are labels on the envelope, one Name: value per line. Names ignore case, so content-type and Content-Type are the same header.
• A blank line marks the end of the headers.
• The body, if there is one, is the content of the letter. Content-Length says exactly how many bytes it has, so the server knows where the message stops.

Every line ends with two invisible characters, carriage return and line feed, written \r\n. They go back to typewriters: return to the start of the line, then move down one. So the blank line is really \r\n\r\n.

raw = "GET /campaigns?id=7 HTTP/1.1"
method, target, version = raw.split(" ")
print(target)

import json

body = json.dumps({"name": "Spring", "budget": 500}).encode("utf-8")
head = (
    "POST /v1/campaigns?dry_run=1 HTTP/1.1\r\n"
    "Host: api.gradient.test\r\n"
    "Content-Type: application/json\r\n"
    f"Content-Length: {len(body)}\r\n"
    "\r\n"
)
raw = head.encode("ascii") + body
print(raw)
print(len(body), "body bytes;", len(raw), "bytes in total")

The response

The server answers in the same shape. Only the first line differs. It's the status line: the version, a three-digit status code, and a short phrase for people to read:

HTTP/1.1 201 Created
Content-Type: application/json
Content-Length: 69
Location: /v1/campaigns/cmp_107

{"id": "cmp_107", "name": "Spring", "budget": 500, "status": "draft"}

The code is the headline. Anything in the 200s means it worked, the 400s mean the request had a problem, and the 500s mean the server did. Here, 201 Created says a new campaign now exists, and the Location header says where to find it.

Try it below. The widget sends real requests to the practice API and shows the raw request, the status, the headers and the body.

URLs and query strings

A web address packs several parts into one line, the way a postal address packs in a name, a street and a city:

https://api.gradient.test:443/v1/products?category=home&q=caf%C3%A9#reviews
scheme  host              port path         query                      fragment

• The scheme says how to talk: here, HTTPS.
• The host and port say which server, and which door.
• The path says what you want on that server.
• The query, after the ?, adds details as key=value pairs, joined by &.
• The fragment, after the #, never reaches the server. Browsers keep it for themselves.

That creates a small problem. ?, &, =, # and / are punctuation in a URL, and spaces aren't allowed at all. So what if your search text contains one of them? It has to travel in disguise. That's percent-encoding: each byte becomes % plus two hex digits (hex is a way of writing numbers using 0 to 9 and a to f).

For example, the character & is byte number 26 in hex, so it travels as %26. A space is %20, or + in query strings. And é takes two bytes in UTF-8, the usual way of turning text into bytes, so it becomes %C3%A9.

In plain words: characters that have a job in a URL get swapped for a code, so they can travel as ordinary data.

Don't do this by hand. Python's urllib.parse gets it right:

from urllib.parse import urlsplit, urlencode, parse_qs, quote

url = "https://api.gradient.test/v1/products?category=home&q=caf%C3%A9#reviews"
parts = urlsplit(url)
print(parts.scheme, parts.hostname, parts.path, parts.fragment)
print(parse_qs(parts.query))

print(urlencode({"q": "tents & tarps", "max_price": 200}))
print(quote("reports/Q3 final.pdf"))

parse_qs returns lists, because a key can repeat: ?tag=a&tag=b is legal and common.

Newer versions, same meaning

So far you've read HTTP/1.1: text, one request at a time on each connection. Two newer versions change how the letter is carried, and leave what it says alone.

• HTTP/2 packs the same requests into compact binary pieces called frames. Many requests can share one connection at once, like several conversations on one phone line.
• HTTP/3 runs over a newer transport called QUIC, which recovers from lost packets faster. That helps most on mobile networks.

Methods, paths, headers, status codes and bodies mean exactly the same in all three. Learn them once in readable HTTP/1.1, and you understand all of them.

Note: The academy's Python can't open raw network connections, so lessons work with the bytes directly. On your own machine, Python's socket module sends exactly these bytes over TCP.

Details that cause real bugs

These four cause most of the strange failures you'll meet.

Header names ignore case. Content-Type and content-type are the same header, and HTTP/2 sends them all in lowercase. A good library handles this for you, but your own dictionary won't: headers["Content-Type"] fails if the server sent lowercase. So never compare header names with a plain ==.

Content-Length counts bytes, not characters. The word café has 4 characters and 5 bytes in UTF-8.

text = "café"
print(len(text), len(text.encode()))

Send the wrong length, and the other side either waits for bytes that never come, or cuts your message short. Encode first, then measure.

Query strings need encoding. A search for boots & coats pasted into a URL breaks, because & separates parameters and spaces aren't allowed. Use urllib.parse.urlencode, or let your HTTP library do it.

The part after # never reaches the server. The browser keeps the fragment to itself. If you've wondered why your server logs never show it, that's the reason. It's also why apps like this one can use it for pages inside the browser, without the server knowing.

Reading a failed request

When an API call fails strangely, look at the raw bytes before you look at your code. It's like checking the envelope before you reread the letter. Most HTTP clients have a debug option that prints them. You'll often spot the problem straight away: a missing header, a body that isn't what you thought, or a URL encoded twice. HTTP is plain text underneath, which makes that kind of check possible.

Tools to know

curl is a small command-line program that sends a request and prints the reply. Add -v, for verbose, and it shows the headers going both ways. It's often the fastest way to see what's really being sent.