llms.txt vs robots.txt vs sitemap.xml
Three root-level files, three different jobs. Here's how they fit together.
| File | Audience | Purpose |
|---|---|---|
robots.txt | All crawlers | Permission: which paths may be fetched |
sitemap.xml | Search engines | Discovery: a complete list of indexable URLs |
llms.txt | AI assistants & agents | Understanding: a curated, described subset of what matters |
They work together
Our crawler reads robots.txt first and never fetches disallowed paths. It then uses your sitemaps to discover pages, prioritises the important ones (home, about, products, services, docs, policies), and writes an llms.txt that describes them.
A common mistake
Blocking AI crawlers in robots.txt while publishing llms.txt sends mixed signals. Decide which AI tools may read your site, set robots.txt accordingly, and use llms.txt to make what you allow easy to understand.