Scraping to feed a RAG

To get public documentation into a RAG — a semantic search that answers while citing your sources — the real work isn’t the model: it is turning pages into clean text, chunking it by section, and recording where each piece came from. If the text goes in dirty, no choice of model fixes it, and the answers will be poor even when retrieval works.

This guide walks that chain step by step, and it’s the same whether you end up using a service or building it yourself.

1. Decide what goes in before collecting anything

The temptation is to crawl the whole domain. It’s the mistake that costs the most: you pull in the nav menu, the footer, the cookie notice and three hundred listing pages, and then your search returns fragments of the privacy policy when you ask about the API.

Start from a short, specific list. The site’s sitemap.xml is the best starting point: it’s the list of what its owner considers real content. Filter by prefix — only /docs/, only /blog/ — and read the titles before accepting anything.

Rule: a RAG with fifty good pages beats one with five thousand mediocre ones. Noise doesn’t dilute, it competes.

2. Convert to Markdown, not plain text

A raw innerText destroys exactly what you need most later: where one section ends and the next begins.

Markdown preserves headings, lists and tables, and that lets you chunk by structure in the next step instead of by character count. It is the difference between a fragment that stands on its own and one that starts mid-sentence.

Firecrawl does precisely this: hand it a URL or a site and it returns clean Markdown with the menu and footer already stripped. That’s what saves writing a custom scraper per site, which is the part that doesn’t scale. If you build it yourself, the equivalent piece is a main-content extractor ahead of the Markdown converter.

3. Chunk by section, not by character count

Chunking is where quality is won or lost, and most tutorials get it wrong.

Cutting every 1,000 characters is easy and produces fragments that start and end mid-idea.

Cutting by heading — each ## is a chunk — produces fragments that answer a complete question, which is exactly what you want to retrieve. With two practical adjustments: if a section runs long, split it by paragraph; if it’s two lines, merge it with the next.

And carry the context: every chunk should be prefixed with the page title and the heading hierarchy containing it. A chunk starting “X Documentation › Authentication › Tokens” retrieves far better than one starting “To do this, use the following value”.

4. Store the source with every chunk

This is the step that separates a useful RAG from one nobody trusts.

Each fragment is stored with its exact URL, its title and the date you collected it. Without that you cannot cite the source, and a RAG that doesn’t cite is indistinguishable from a model making things up: the reader has no way to check anything.

The date matters more than it looks. Year-old documentation answering about an API that changed is worse than not answering.

5. Choose where to search

There is less of a decision here than it seems:

• Postgres with pgvector. The sensible default for small and medium projects: one less piece to maintain, and you already have the database. Supabase and Neon ship it, and we compare them in free-tier databases.
• A dedicated vector database. Qdrant has a free tier and earns its place when volume or latency stop fitting.

And a piece of advice nearly everyone skips: try plain old text search first. For technical documentation full of proper nouns and codes, Postgres’s lexical search does better than people expect, and combining it with vector search usually beats either on its own.

6. Re-collect, and notice what changed

A RAG built once and never updated ages silently: it keeps answering confidently about documentation that no longer exists.

Store a hash of each page’s content. On the next pass you only reprocess what changed: you spend less and, more importantly, you get a list of what moved. Schedule the run and alert on it having run: Healthchecks.io warns you when a job stops checking in, which is how these things fail — silently, returning zero.

All the pieces and their free plans are in free scraping tools. And before collecting from someone else’s site, the four questions in legal scraping: public documentation is the easy case, but it’s worth reading the licence anyway.

Frequently asked questions

How big should each chunk be? Less important than where you cut. With heading-based chunking the size takes care of itself and usually lands between 200 and 800 words. If you must pick a fixed number because the content has no structure, start small with overlap and measure before raising it.

Do I need a vector database? For most projects, no. Postgres with pgvector handles tens of thousands of chunks well and saves you another service. A dedicated one is justified by volume or latency, not by default.

Why does my RAG answer badly when retrieval finds the right documents? Almost always the chunking. If the retrieved fragment starts mid-explanation or lacks its section’s context, the model receives something that doesn’t stand on its own. Before changing model, print the fragments you retrieve and read them: if they aren’t enough for you, they aren’t enough for the model.

Can I ingest an entire site at once? You can, and it almost always makes the result worse. The menu, the footer and listing pages generate fragments that compete with the good content. Filter by URL prefix and review the list before processing.

How often should I re-collect? It depends on how fast the source changes: documentation for an active product, every week or two; stable content, monthly. What matters isn’t the exact frequency, it’s that it exists and that it tells you if it ever stops running.