Crawl settings & schedule
Control which pages are included and how often they refresh.
What we crawl
We start from your homepage and your sitemaps, follow internal links up to the configured depth, and only fetch pages that robots.txt allows for BlogRainbowBot (or *). We honour Crawl-delay (up to 10 seconds), skip pages marked noindex, and never log in or bypass access controls.
Default exclusions
Login, registration, account, cart, checkout, admin and search pages, internal routes (/wp-json, /api…), tag/author archives, URLs with query parameters, and non-HTML files.
Your settings
- Maximum pages (limited by your plan) and maximum depth
- Include / exclude patterns: simple wildcards such as
/blog/*or*/print/* - Page types: products, blog, static pages
- AI descriptions on/off
- JavaScript rendering for pages that are empty without JavaScript (Starter plan)
Schedule
Choose daily, weekly (default: Monday 03:00 in your timezone), monthly or manual. Scheduled crawls only run for verified websites. Unchanged pages are detected with HTTP caching headers and content hashes, so repeat crawls are fast.