Scraping to feed a RAG
A step-by-step guide to turning public documentation into clean text, chunking it by section, and putting it into a semantic search without writing a custom scraper.
Blog
Writing down what maintaining the directory teaches us
Reviewing 115 services one by one turns up things the pricing page never says. They are written down here, so they save you an afternoon.
16 articles Page 1 of 2
Legal scraping: what you can and can't
The four questions that decide whether a scrape is defensible: public data, terms of service, personal data and copyright. Plus what never to do.
Web scraping: where to start
What web scraping is, when it beats an official API, and the five decisions that determine whether your scraper lasts a month or breaks on Tuesday.
Free scraping tools that hold up
The six pieces a scraper needs — HTTP client, parser, browser, queue, storage and alerts — and which options have a genuinely free plan for each.
How to get AI assistants to cite your site
ChatGPT, Perplexity and Google's AI summaries cite specific sources. What makes them pick yours: direct answers, HTML without JavaScript, structured data and an llms.txt.
Free SEO tools for a small project
The seven pieces you need to measure and fix the SEO of a small project without paying anything, and what to check in each one's free plan before you set it up.
hreflang: SEO for a bilingual site
How to get Google to index both versions of your site instead of treating one as a copy of the other. With the canonical mistake that erases an entire language from the index.
Structured data: what to mark up and what not
JSON-LD without the hype: which types actually do something, how to stop the schema promising things the page doesn't show, and why it doesn't raise rankings but does raise clicks.
sitemap.xml and robots.txt done right
A step-by-step guide to a sitemap that lists exactly what is public and a robots.txt that blocks only what is private. With the three failures that break nothing and drop you from the index.
The directory is the other half of this
115 services with a genuinely free plan, each one with its limits spelled out. No sign-up needed to start looking.