No crawl, no citation
Every AI visibility tactic assumes one thing: that an AI crawler can fetch your pages. GPTBot trains OpenAI models, OAI-SearchBot fetches pages live for ChatGPT answers with citations, ClaudeBot and PerplexityBot do the same for their ecosystems. If your robots.txt blocks them — often inherited from a years-old 'block all bots' rule aimed at scrapers — no amount of content optimization matters. The models simply never see the pages.
The five tokens that matter most
You don't need to audit all 600+ known crawlers. Start with five: GPTBot and OAI-SearchBot (OpenAI training + live answers), ClaudeBot (Anthropic), Google-Extended (controls Gemini and AI Overviews training use), and PerplexityBot (the pages Perplexity cites). If all five can fetch your site, you've covered the engines behind the vast majority of AI answers your buyers see.
How to check in under a minute
Open yourdomain.com/robots.txt and search for those five names. A 'Disallow: /' under a crawler's User-agent block means a total block; path-specific Disallows (like /admin or /search) are usually fine and intentional. Absence of a robots.txt file entirely means everything is crawlable by default — which is also an answer. Our free AI Crawler Check at /tools/robots does this for all 17 major AI crawlers automatically and grades each one allowed, partial, or blocked.
What to actually change
Unblock the AI crawlers you want learning from you, keep blocking what costs you money (aggressive scrapers, bandwidth hogs). The change is usually three lines: explicit 'Allow: /' or simply removing the AI tokens from a wildcard Disallow. Then verify — re-run the check — and move on to the work that actually differentiates: content worth citing. Crawlability is step zero, not the strategy.
See this on your own brand
Run the free check — same scoring the dashboard uses, no account needed.
Run free check