Legal scraping: what you can and can't

Whether a scrape is defensible comes down to four things, and none of them is “can you technically do it”: whether the data is genuinely public, what the terms of service say, whether personal data is involved, and who owns the content. Extracting public, non-personal data without harming the source service and without republishing someone else’s content is the quiet zone. Outside it, the answer stops being general.

A necessary warning before going on: this is not legal advice. It is a map of the questions worth asking, written by people who write code, not by lawyers. The rules vary by country and by case; if the project is commercial or touches people’s data, talk to someone who practises.

1. Is the data genuinely public?

“Public” doesn’t mean “reachable if I try hard”. It means anyone can see it without identifying themselves.

• Public: a product page, an article, a listing that renders without signing in.
• Not public: anything behind a login, a paywall, or a form that requires accepting conditions.

That boundary matters far more than it looks. Accessing with credentials that aren’t yours, or getting around a control deliberately placed to prevent automated access, stops being “reading a page” in most legal systems and lands in unauthorised-access-to-a-computer-system territory. It is the line not to cross, not even with good intentions.

With your own account, extracting your own data, the case is very different and usually fine.

2. What do the terms of service say?

Almost all terms of service forbid automated access. The interesting question isn’t whether they do — they do — but how much weight that carries.

Simplifying heavily: terms of service are a contract, and a contract binds whoever accepts it. If you created an account and ticked the box, you accepted them. If you never signed in and simply requested a public URL, the argument that you’re bound by them is considerably weaker, and several courts have seen it that way.

What is certain in every case: breaching them can cost you the block and, where there’s a commercial relationship, the account. Being hard to enforce doesn’t make it a good idea.

And robots.txt: it is neither a contract nor a technical barrier, it’s a request. Ignoring it is rarely illegal in itself, but it is the first exhibit someone will produce to argue you were acting in bad faith. Respecting it is cheap. How it’s written, in sitemap.xml and robots.txt.

3. Is there personal data?

This is where it gets genuinely serious, and where most people land in trouble without noticing.

Personal data being public doesn’t take it out of the regulation. A name, an email, a profile or a signed review are still personal data even in plain sight, and in Europe the GDPR applies regardless. Collecting them at scale requires a legal basis, informing people, and honouring their rights — including erasure. Latin America has equivalent rules: in Colombia, Ley 1581.

Three practical rules that cut the risk sharply:

• Don’t collect personal data you don’t need. Most projects don’t need it: they want prices, availability or text.
• If personal data slips in by accident, drop it at processing time, not “I’ll clean it later”.
• Harvesting emails to send marketing isn’t a grey area. It is precisely what the regulation targets.

4. Whose content is it?

An article, a photo or a product description is someone’s work. Extracting them to analyse internally is one thing; republishing them is quite another.

The practical difference:

Use	Risk
Analysing, counting, comparing internally	Low
Keeping a copy to reprocess	Low
Showing a short excerpt with attribution	Moderate
Republishing the full text	High
Republishing it in competition with the original	Very high

Harm to the service counts too: a scraper that saturates a small server can be treated as abuse regardless of what it extracts. Good technical manners — rate limiting, caching, identifying yourself — aren’t just politeness, they’re part of the argument that you acted reasonably. They’re in web scraping: where to start.

What never to do

Four things that turn a debatable project into an indefensible one:

1. Getting around CAPTCHAs or anti-bot systems. That is an explicit no. Dodging it removes any good-faith defence.
2. Using credentials that aren’t yours, even if they were lent to you.
3. Collecting personal data at scale with no legal basis and no way for anyone to exercise their rights.
4. Taking down the source site. Even if the data is public, someone else pays for the bandwidth.

Write it down

If the scraping feeds something that will last, put a paragraph in the repository: which site, what data, what for, how often, and who to email if someone complains. It takes five minutes and it turns an awkward conversation into a two-line reply.

And put a contact in the User-Agent. An admin who can email you will rate-limit you; one who can’t will block you.

Frequently asked questions

Is web scraping illegal? No, there is no general prohibition on scraping. What can be illegal is something specific you do with it: accessing what’s protected, processing personal data without a legal basis, republishing someone’s work, or damaging the service. Which is why the useful question isn’t “is scraping legal?” but “what am I extracting, from where, and what for?”.

Is respecting robots.txt enough to be safe? No. robots.txt says nothing about personal data or copyright, which are the two serious fronts. Respecting it helps demonstrate good faith, but it doesn’t cover the other three questions in this article.

Can I scrape a site to train a model? It is the least settled ground of all right now, with open litigation and criteria that vary by country. For internal, experimental use the risk profile differs from a commercial product. If it’s the latter, this is exactly the case where you should ask before you build.

What if the site blocks me? That is their right, and the useful answer is almost never more disguise. Slow down, identify yourself, and if the project warrants it, email whoever maintains it: it often ends in agreed access or an API you didn’t know existed.

Can public-body data be used freely? It usually carries an open-data licence that allows it, sometimes requiring attribution. Read it: “public” and “reusable with no conditions” are not the same, and many portals also offer a direct download that saves you scraping at all.