Glossary · Phase 0 · Tools
robots.txt
A file at the root of a website that tells automated programs which paths to stay away from and how fast to go.
Try it now
When a site offers a proper way in, use it. Scraping is for public data with no better route, collected gently.
Learn it properly
Scraping web pages · Beautiful Soup, one record per item, pagination and robots.txt. About 45 minutes.
Other terms from the same lesson
- web scraping: Having a program read web pages and pull data out of their HTML, when there's no API to ask.
Never trained an AI?
Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account.