Skip to content
BlogRainbow AI Ready

How it works

From your URL to a verified, published llms.txt

A careful pipeline that respects your site and never invents facts.

  1. Step 1

    Add your website

    Enter your URL. We check robots.txt, find your sitemaps, detect your platform and produce a first readiness report.

  2. Step 2

    Verify ownership

    Add a DNS TXT record, upload a small file, or paste a meta tag. You can preview your files before verifying.

  3. Step 3

    We crawl and generate

    Allowed pages are fetched politely, converted to clean Markdown, and turned into llms.txt, llms-full.txt and JSON files.

  4. Step 4

    Publish and stay current

    Files go live on your hosted URL (or your domain via the plugin) and refresh on your schedule.

Inside each crawl

  1. 1.DiscoverWe read robots.txt and your sitemaps, then rank URLs so the pages that matter most (home, about, products, services, docs, policies) come first.
  2. 2.Fetch politelyOne page at a time, honouring your Crawl-delay. Unchanged pages are recognised with HTTP caching headers and skipped.
  3. 3.ExtractEach page becomes clean Markdown. Structured data gives us products, prices, FAQs and contact details exactly as published.
  4. 4.DescribeA local AI model writes short page descriptions. A fact guard rejects any sentence with numbers, prices, emails or links not found on the page.
  5. 5.Generate and validatellms.txt, llms-full.txt, Markdown and JSON files are built and validated: structure, links, encoding, sizes and schemas.
  6. 6.Publish and verifyValid files are published atomically as a new version. We then fetch the public URL to confirm it serves exactly what we published.

More details: crawl settings, publishing options, and how we keep AI text factual.