Web scraping: where to start

Web scraping is reading a web page with a program and pulling structured data out of it. You use it when the information is public but there is no clean way to query it: a catalogue with no API, documentation you want inside your own search, prices scattered across a hundred pages. The first decision, and the one that saves the most time, is checking whether you need to scrape at all.

This guide covers that decision and the four that follow — the ones that separate a scraper that holds up from one that breaks with the first layout change.

Do you actually need to scrape?

Before writing a line, three checks in this order:

Is there an official API? Almost always the right answer. An API gives you stable data with a contract, and a CSS change won’t take your pipeline down. Even a paid one is usually cheaper than maintaining a scraper.

Is there a feed? RSS, a sitemap.xml, a JSON endpoint the page itself uses underneath. A great many sites serve their data as JSON for their own frontend: open developer tools, look at the network tab, and check before you parse any HTML.

Is there a download? Open data, a CSV, a dump. Public bodies publish far more than people expect.

If all three are no, then yes: scraping. And that is where the decisions start.

1. Do you need a browser, or is HTML enough?

This is the decision with the biggest impact on cost.

A plain HTTP request. You fetch the URL, get HTML, parse it. Fast, cheap, and it works with any server-rendered site. This is what you should try first, always.

A headless browser. You load the page in a real browser, run its JavaScript and read the result. Necessary when the content only appears after hydration. It costs between ten and a hundred times more in time and memory.

The check takes thirty seconds: curl the URL and look for your data in the response. If it’s there, you don’t need a browser. If it isn’t, look for a JSON call underneath before resigning yourself to one.

2. How to point at the data so it doesn’t break

A scraper breaks because a selector stops existing. Resilience is decided here:

• Most stable: structured data. Many pages already carry JSON-LD with the price, author or date — precisely because they put it there for Google. Read it from there rather than from rendered HTML: it is declared content, not layout. We cover it from the other side in structured data: what to mark up and what not.
• Stable: semantic attributes and microdata, <time datetime>, <meta property="og:…">.
• Fragile: utility classes and classes generated by the CSS compiler. A div.css-1x2y3z is a time bomb.
• Worst: positions. “The third div in the second section” lasts until the next deploy.

Rule of thumb: if the selector describes what the data is, it holds; if it describes where it sits, it doesn’t.

3. Good manners, which is also what works

An aggressive scraper ends up blocked, and rightly so: you are costing somebody money. What a well-behaved client does:

• Read robots.txt and respect it. It is a request, not a technical barrier, and ignoring it is the first sign you are acting in bad faith. How it’s written is in sitemap.xml and robots.txt.
• Identify yourself in the User-Agent, with a name and a way to reach you. An admin who sees odd traffic and can email you will rate-limit you; one who can’t will block you.
• Limit your rate. One request per second is far more than you usually need. Go serial before you go parallel.
• Honour 429 and Retry-After. If you’re asked to wait, wait, with exponential backoff.
• Cache. Most scrapers re-request the same thing day after day. Store the response and use If-Modified-Since.

What this guide deliberately does not cover: getting around CAPTCHAs, signing in with someone else’s credentials, or evading anti-bot systems. That is where the real legal trouble starts, and we go into it in legal scraping: what you can and can’t do.

4. Where you put what you extract

A scraper produces dirty, duplicated data. Two pieces of advice that save a rewrite:

Store the raw HTML, not only what you extracted. The day you find you were reading a field wrong, being able to reprocess without re-requesting the pages is worth its weight in gold — and it spares the source site the traffic.

Give every row a stable key. The normalised canonical URL usually does. Without it, the second pass duplicates everything.

For where to put it, Postgres covers 90% of cases; we compare the options in free-tier databases.

5. What happens when it fails

It will fail. The question is whether you find out:

• Count what you extract. If yesterday you got 400 rows and today 12, something changed even though the job finished without error.
• Fail loudly on an empty required field. A null price saved silently contaminates the whole dataset.
• Store the status code. A whole batch of 403s is a block, not a parsing bug.

Where to go next

If the project is small, start with HTTP requests, a parser and one table. If you need to turn pages into clean text for a language model, services already do that and save you all of the above: it’s in scraping to feed a RAG. And the free-plan options are in free scraping tools.

Frequently asked questions

Is web scraping legal? It depends on what, where and how, and there is no single answer. Extracting public, non-personal data for your own use is generally considered acceptable; the conflicts arise around terms of service, personal data and copyrighted content. We go into it in the dedicated article, and for a specific case you need a lawyer.

Which language is best for scraping? The one you already use. Python has the largest ecosystem and Node fits better if you’ll need a headless browser. Language choice is the factor that least affects whether the scraper holds up.

How often does a scraper break? Every time the site changes its layout, and that comes with no warning. With semantic selectors or structured data you can go months; with generated classes, weeks. That is why point 5 matters more than point 2: not preventing breakage, but finding out the same day.

Can I scrape a site that requires signing in? With your own account and for your own data, usually yes. With credentials that aren’t yours, no: besides breaching the terms of service, in many countries that lands in unauthorised-access territory.

Is a headless browser always more reliable? No, and it is more expensive and more fragile to operate. It only helps when the content needs JavaScript to exist. If the data is already in the first response’s HTML, a browser only adds latency and memory.