Skip to content
BlogRainbow AI Ready
← Blog

llms.txt vs robots.txt vs sitemap.xml

Three root-level files, three different jobs. Here's how they fit together.

FileAudiencePurpose
robots.txtAll crawlersPermission: which paths may be fetched
sitemap.xmlSearch enginesDiscovery: a complete list of indexable URLs
llms.txtAI assistants & agentsUnderstanding: a curated, described subset of what matters

They work together

Our crawler reads robots.txt first and never fetches disallowed paths. It then uses your sitemaps to discover pages, prioritises the important ones (home, about, products, services, docs, policies), and writes an llms.txt that describes them.

A common mistake

Blocking AI crawlers in robots.txt while publishing llms.txt sends mixed signals. Decide which AI tools may read your site, set robots.txt accordingly, and use llms.txt to make what you allow easy to understand.