Scraping web pages with Beautiful Soup
Why this matters
Sometimes the data you need is only on a web page: a competitor's prices, a list of events, a government table. Scraping turns that page into data. It's also easy to do badly: code that breaks when the page changes, that hammers a small site, or that takes what it shouldn't. This lesson teaches the technique, and the habits that keep your scrapers reliable and polite.
- Check for an API and for structured data in the page before scraping its HTML
- Parse a page with Beautiful Soup and pull out one record per item with CSS selectors
- Clean scraped text into numbers, and turn relative links into full addresses with urljoin
- Follow pagination politely: read robots.txt, skip what it forbids and wait between requests
Before you scrape: three questions
Scraping HTML is the last resort, not the first:
- Is there an API? Many sites offer one, often linked at the bottom of the page as "Developers" or "API". An API gives clean JSON, is meant to be used by programs, and doesn't break when the page's design changes.
- Is the data in the page as structured data? Many product and event pages carry a
<script type="application/ld+json">block for search engines. It holds the name, price and rating as JSON, and it's far more stable than the visible layout. - May you take it? Read the site's terms and its
robots.txt. Never scrape pages behind a login, and never collect personal data you have no right to.
The practice server hosts a small shop website for this lesson at https://api.gradient.test/shop/, with a deals section that runs over three pages.
Fetch and parse
requests fetches the page as text, and Beautiful Soup parses that text into a tree you can search:
import requests from bs4 import BeautifulSoup reply = requests.get("https://api.gradient.test/shop/deals", params={"page": 1}) print(reply.status_code, reply.headers["content-type"]) soup = BeautifulSoup(reply.text, "html.parser") print(soup.title.get_text()) cards = soup.select("article.deal") print(len(cards), "cards") first = cards[0] print(first.prettify()[:420])
"html.parser" is the parser built into Python; Beautiful Soup can also use faster ones like lxml. Printing one card with prettify() is the most useful thing you can do when you start. You see exactly which tags and classes hold the name, the price and the link, so you can choose selectors for them. Your browser's "Inspect" tool shows the same thing for a real site.