sitemap.xml and robots.txt done right

A good sitemap.xml contains exactly the public URLs you want indexed, once each, in their canonical form. A good robots.txt blocks what sits behind an account and nothing else. It sounds obvious, and yet both files corrupt themselves over time: they break no test, throw no browser error, and go stale with nobody looking.

This guide goes step by step, with the specific failures we hit ourselves and how to catch them.

1. Decide what is public before writing any XML

Before the markup, a list. Public is what you want a search engine to show; private is everything else: sign-in, sign-up, anything behind an account, the API, and any URL that is secret by obscurity — a share link with a token, for instance.

That last case deserves attention: if a search engine indexes your secret link, it stops being secret. Not linking to it isn’t enough; it has to be blocked explicitly.

Write that list in exactly one place in the code and have both the sitemap and the robots.txt read from it. If it lives twice, one day one copy gets updated and the other doesn’t.

2. Generate the sitemap, don’t hand-write it

A hand-written sitemap has a half-life of about two months. Generate it from the same source the site uses to render its links.

There is an important trap on server-rendered sites: sitemap generators discover static routes at build time. If all your pages are dynamic, they discover none, and the sitemap comes out empty or partial with no warning. That is exactly what happened to us: two pages went from static to dynamic and disappeared from the sitemap without breaking a single test. The fix was to declare the route list in a shared module and hand the generator the whole thing.

For anything born after the build — community content, articles, anything in a database — you need a separate sitemap generated per request that actually queries the source.

3. One URL, one form

The quietest mistake of all: listing the same page twice in different forms.

https://example.com/about
https://example.com/about/

To a search engine those are two URLs with identical content, and they contradict the canonical, which can only point at one. We ended up with six duplicate pairs like that and nothing flagged it, because checking “is this page in the sitemap?” passes either way.

Pick one form — with or without the trailing slash — and filter the other out when generating. Make the canonical, the internal links and the sitemap all say the same thing.

4. With more than one language, declare the alternates

When the same page exists in two languages, each sitemap entry must declare both with xhtml:link:

<url>
  <loc>https://example.com/blog/my-post</loc>
  <xhtml:link rel="alternate" hreflang="es" href="https://example.com/blog/my-post" />
  <xhtml:link rel="alternate" hreflang="en" href="https://example.com/blog/my-post?lang=en" />
</url>

Without this, the secondary version may never get indexed at all. The details are in hreflang: SEO for a bilingual site.

5. Write robots.txt from the same list

robots.txt is not a security mechanism: it is a request that serious crawlers honour. Anything behind an account is protected by authentication, and blocked here.

A structure that works:

User-agent: *
Allow: /
Disallow: /app
Disallow: /api
Disallow: /s/

Sitemap: https://example.com/sitemap-index.xml

Three details people skip:

• The sitemap URL must be absolute, with the domain.
• If you have several sitemaps, declare them all. One declared and three orphaned is the usual pattern once a site grows.
• Generate the file rather than shipping it static if the domain differs between environments. A production robots.txt with the staging domain inside is a classic.

6. Decide what you do about AI crawlers

This is a new decision and deserves thought rather than inheritance. Not all AI crawlers do the same thing:

• The ones that read to answer and cite the source are the assistants with live search. They bring real visits and appear as a link in the answer.
• The ones that only harvest for training consume bandwidth and return nothing measurable.

You can treat them differently, agent by agent. We give the first group the same access as Google — the content is already public, there is nothing to gain by closing it — and block the second. We also added an llms.txt, the site summarised in Markdown so a model doesn’t have to crawl a hundred pages of HTML; that’s in how to get AI assistants to cite your site.

7. Build a check that fails on its own

None of the above survives six months unsupervised. A script that crawls your own site and exits with code 1 when something is off is worth more than any audit.

What it should check, at minimum:

• That robots.txt responds, blocks the private prefixes and declares every sitemap.
• That the sitemap exists and has no duplicate trailing-slash URLs.
• That every public page is in the sitemap and no private one is.
• That no public page carries noindex, and private ones do.
• That there are no broken internal links.

Hang it off your CI and the problem is over. It is point 8 of technical SEO for developers, and the best return on the list.

Frequently asked questions

Does a sitemap improve rankings? Not directly. It helps what you have get discovered, especially new or poorly linked pages. A page already crawled normally doesn’t rank better for being listed; one that isn’t being discovered can take far longer to appear without it.

How many URLs fit in one sitemap? Fifty thousand URLs or fifty uncompressed megabytes, whichever comes first. Past that they are split and declared together in a sitemap index, which any decent generator does by itself.

Should I include the blog’s paginated pages? You can, and it is the consistent choice if your principle is that the sitemap is the complete list of what’s public. What you must not do is give them all the canonical of page 1: each listing page is different content and its canonical points at itself.

Does Disallow in robots.txt remove a page from the index? No, and it is the most expensive confusion in technical SEO. Disallow prevents crawling, not indexing: a blocked URL that receives links can still show up in results, without a description. Removing it from the index needs noindex, and for that the crawler has to be able to read the page — so don’t block it at the same time.

How often should the sitemap be updated? Never by hand. If it is generated from the real source, it updates when the content changes and there is nothing to remember. That is the entire point of the exercise.