Free scraping tools that hold up

Building a scraper that holds up takes six pieces, and nearly all of them have a genuinely free option: something to fetch the page, something to parse it, a browser for when HTML isn’t enough, a queue so you’re not serial forever, somewhere to store the results, and something to tell you when it breaks. None of them is expensive on its own — the expensive part is not knowing which ones you need.

Here is the list, in the order you assemble it. Each service’s concrete limits are on its page in the directory, because they change often and would go stale here.

1. The HTTP client: what you already have

The first piece is free in every language and needs nothing installed: fetch in Node and the browser, requests or httpx in Python, curl in the terminal.

What matters isn’t the library, it’s three settings: a timeout (without one, a slow server hangs your whole job), retries with exponential backoff, and a User-Agent that identifies you. That covers 80% of server-rendered sites.

To try requests before writing code, Hoppscotch is a browser-based API client with a free plan.

2. The parser: from HTML to data

Here too the language ecosystem decides, and everything good is open source: BeautifulSoup and lxml in Python, Cheerio in Node.

The advice that saves the most work isn’t about libraries: before parsing HTML, look for the page’s JSON-LD. A great many sites carry the price, date and author declared in an application/ld+json block because they put it there for Google, and reading it from there is vastly more stable than chasing CSS classes.

3. The headless browser: only when you need it

Playwright and Puppeteer are free and open source, and they are the answer when content only exists after JavaScript runs.

Two warnings:

• Check first. Fetch the URL with curl and look for your data. If it’s in the HTML, a browser only adds latency and memory.
• It’s expensive to operate. Each instance eats hundreds of megabytes. In a serverless function it often doesn’t fit.

If you need a browser but don’t want to run the infrastructure, hosted services exist; there the free plan is usually counted in browser-minutes per month, and it goes fast.

4. Scraping APIs: paying not to maintain it

These are services you hand a URL and get resolved content back. They save you the browser, the retries and much of the maintenance.

Firecrawl is the directory’s free-plan option: it turns a page or a whole site into clean Markdown, which is exactly what you want if the destination is a language model. We go into it in scraping to feed a RAG.

For narrower cases there are small APIs that spare you a whole scraper: OpenGraph.to returns the title, description and image of any URL, and Abstract API offers a set of utilities each with its own free quota. If all you need is link metadata, don’t build a scraper.

In any of their free plans, check credits per month and what counts as a credit: on some, a JavaScript page costs five.

5. The queue: so you’re not serial forever

Past a hundred URLs you need something that distributes the work, retries failures, and doesn’t fire everything at the same domain at once.

For a small project, a table with a status column (pending, done, failed) and a loop is enough and adds no dependencies. When it grows, Upstash offers Redis and queues on a per-request free plan, which suits occasional runs, and Pipedream can orchestrate the batch without a server.

A rule almost nobody applies, and it prevents blocks: limit concurrency per domain, not in total. Ten simultaneous requests spread over ten sites bother nobody; ten at once against one site do.

6. Storing, and finding out when it breaks

For storage, Postgres covers nearly everything and is the most portable: Supabase and Neon have free plans, and we compare them in free-tier databases. Store the raw HTML too: the day you find you were reading a field wrong, being able to reprocess without re-requesting the pages pays for itself.

And the piece almost nobody sets up: alerts. A scraper fails silently, returning zero rows without throwing anything. Healthchecks.io warns you when a scheduled job stops checking in, and Sentry collects the exceptions. Full category in monitoring.

What isn’t worth getting for free

Residential proxy services and CAPTCHA-solving services. It isn’t about price: they are exactly the line where scraping stops being reading a public page and becomes dodging someone who told you no. It’s covered in legal scraping: what you can and can’t do.

If a site blocks you systematically, the useful answer is almost never more disguise: it’s slowing down, identifying yourself, or emailing whoever maintains it.

Frequently asked questions

Can you scrape without paying anything? For a small project, easily: the HTTP client, the parser and the browser are open source, and the database and alerts have free plans. The first thing that usually forces a payment isn’t the scraping, it’s storage volume or browser minutes.

Is a scraping API worth it, or should I build my own? Build your own if the site is simple and it’s a few pages; pay if you need a browser, many different domains, or you don’t want to maintain it. The real cost of your own scraper isn’t writing it, it’s fixing it every time the site changes.

What breaks first? Almost always the selector, because the site changed its layout. Which is why it pays to read structured data where it exists and to count how many rows each run extracts: a sharp drop is the signal, and it arrives before any exception.

Do I need proxies? For a normal project, no. If you think you do, what you usually actually need is to go slower, cache and identify yourself. Proxies solve a scale problem most projects don’t have.

Can I scrape from a serverless function? With HTTP requests, yes, and it fits well. With a headless browser it depends on your platform’s memory and execution-time limits; that is one of the things worth checking beforehand, like the rest of the free-plan limits nobody reads.