Scraping web pages

Before you scrape: three questions

Scraping HTML is the last resort, not the first:

1. Is there an API? Many sites offer one, often linked at the bottom of the page as "Developers" or "API". An API gives clean JSON, is meant to be used by programs, and doesn't break when the page's design changes.
2. Is the data in the page as structured data? Many product and event pages carry a <script type="application/ld+json"> block for search engines. It holds the name, price and rating as JSON, and it's far more stable than the visible layout.
3. May you take it? Read the site's terms and its robots.txt. Never scrape pages behind a login, and never collect personal data you have no right to.

The practice server hosts a small shop website for this lesson at https://api.gradient.test/shop/, with a deals section that runs over three pages.

Fetch and parse

requests fetches the page as text, and Beautiful Soup parses that text into a tree you can search:

import requests
from bs4 import BeautifulSoup

reply = requests.get("https://api.gradient.test/shop/deals", params={"page": 1})
print(reply.status_code, reply.headers["content-type"])
soup = BeautifulSoup(reply.text, "html.parser")

print(soup.title.get_text())
cards = soup.select("article.deal")
print(len(cards), "cards")
first = cards[0]
print(first.prettify()[:420])

"html.parser" is the parser built into Python; Beautiful Soup can also use faster ones like lxml. Printing one card with prettify() is the most useful thing you can do when you start. You see exactly which tags and classes hold the name, the price and the link, so you can choose selectors for them. Your browser's "Inspect" tool shows the same thing for a real site.

One container per item

The reliable pattern: select every item's container, then look inside each one for its fields. Each card then gives one complete record, and a missing field affects only that record:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

url = "https://api.gradient.test/shop/deals?page=1"
soup = BeautifulSoup(requests.get(url).text, "html.parser")

def money(text):
    return float(text.replace("$", "").replace(",", ""))

for card in soup.select("article.deal"):
    link = card.select_one("h2 a")
    was = card.select_one(".was")
    print({
        "sku": card["data-sku"],
        "name": link.get_text(strip=True),
        "price": money(card.select_one(".now").get_text()),
        "was": money(was.get_text()) if was else None,
        "url": urljoin(url, link["href"]),
        "sponsored": "sponsored" in card["class"],
    })

• select_one returns the first match or None, so check before using an optional field like the old price.
• card["data-sku"] reads an attribute; card["class"] is a list of the element's classes.
• get_text(strip=True) returns the text without surrounding spaces.
• The link is /shop/products/TE-101, a path without a host. urljoin combines it with the page's address into a full URL you can fetch.

Notice the sponsored card: it's an ad dressed as a deal. A scraper copies whatever is on the page, so deciding what counts is your job.

from urllib.parse import urljoin
base = "https://api.gradient.test/shop/deals?page=1"
print(urljoin(base, "/shop/products/PA-108"), urljoin(base, "?page=2"))

import re

def price(text):
    pass

print(price("$1,095.00"), price(" $46.75 "), price("USD 30"), price("Call for price"))

Following pages, politely

Lists often run over several pages. Find the "next" link and follow it until there isn't one, rather than guessing page numbers. Before fetching anything, read the site's robots.txt: a plain-text file at the root of the site that says which paths automated programs should stay away from. Python reads it for you:

import time
from urllib import robotparser

robots = robotparser.RobotFileParser("https://api.gradient.test/robots.txt")
robots.read()
for path in ["/shop/deals?page=2", "/shop/products/TE-101", "/shop/cart?add=TE-101", "/shop/account/"]:
    print(path, robots.can_fetch("*", "https://api.gradient.test" + path))
print("wait between requests:", robots.crawl_delay("*"), "second(s)")

can_fetch says whether a user agent may fetch a URL; "*" means any program. The practice shop forbids the cart and account pages, and asks for one second between requests. A scraper that clicks every "Add to cart" link would be filling carts and fetching pages it was asked to leave alone.

Polite scraping comes down to a few habits:

• Obey robots.txt, and wait between requests with time.sleep, at least the crawl delay.
• Fetch only the pages you need, and save what you fetch, so a rerun doesn't hit the site again.
• Send a User-Agent header that says who you are and how to reach you.
• Stop if you get errors like 429 ("too many requests"), and never try to get around a block.

When pages change, or need JavaScript

A scraper depends on a page you don't control, and sooner or later the page changes. Plan for it:

• Fail loudly. If a selector finds nothing, that's probably a redesign, not "no deals today". Check the count you expect, and raise an error or send yourself an alert rather than saving an empty result.
• Keep selectors few and meaningful: a class like price or a data-sku attribute survives more redesigns than "the third span in the second div".
• Save what you collect with the time you collected it, as CSV or JSON. You can then rerun your analysis without fetching again, and compare one week with the next.

Some pages arrive almost empty and fill themselves in with JavaScript after loading. requests only sees the empty version, because it doesn't run JavaScript. Open your browser's developer tools, look at the Network tab while the page loads, and you'll usually find the JSON API the page calls. Use that directly. When there's truly no other way, browser automation tools drive a real browser, at the cost of being much slower and heavier.

import json
import requests
from bs4 import BeautifulSoup

page = requests.get("https://api.gradient.test/shop/products/TE-101").text
soup = BeautifulSoup(page, "html.parser")

block = soup.find("script", type="application/ld+json")
data = json.loads(block.string)
print(data["@type"], "|", data["name"])
print("price:", data["offers"]["price"], data["offers"]["priceCurrency"])
print("rating:", data["aggregateRating"]["ratingValue"], "from", data["aggregateRating"]["reviewCount"], "reviews")

shown = soup.select_one(".product .now").get_text()
print("the page shows", shown, "and the data says", data["offers"]["price"])