Troubleshooting crawls
Common reasons a crawl fails or a page is missing.
"robots.txt could not be fetched (…)"
Crawlers must treat an unreachable robots.txt as "disallow everything", so nothing is crawled until it can be read. The reason is shown in brackets:
- HTTP 5xx or 429: your server returned an error for
/robots.txt. Make sure it returns 200 (or 404 if you have none). - SSL certificate problem: your HTTPS certificate could not be verified. If it mentions an incomplete chain, your server does not send its intermediate certificate. Browsers often hide this, but many crawlers and AI agents fail on it. We download the missing certificate automatically when your certificate says where to find it. The proper fix is to install the full chain ("fullchain") on your server or CDN. You can check it with any SSL checker.
- connection failed / timeout: your server or firewall did not answer us. Allow our user agent
BlogRainbowBot.
Then start a new crawl.
A page is missing from llms.txt
Check the crawl log on the Crawl history tab. Common reasons:
- disallowed by robots.txt, or marked
noindex - matched a default exclusion (cart, account, search, parameter URL…) or one of your exclude patterns
- beyond your plan's page limit or the maximum depth
- the page needs JavaScript (enable JavaScript rendering)
Pages come out empty
Your site probably renders content with JavaScript. Enable JavaScript rendering in Crawl settings (Starter plan).
Verification fails
- DNS: wait up to an hour after adding the record and check the record name.
- File: make sure it is served as plain text at the exact path shown, without redirects to a login page.
- Meta tag: it must be on the homepage inside
<head>.
The badge shows "Needs attention"
Open the website's Overview: the reasons are listed there. Run a new crawl or publication check after fixing them.