AI crawler
An automated agent that fetches web content for an AI system: for training corpora, for search and answer indexes, or live at a user’s request. Examples include GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. Each can be allowed or refused in robots.txt.
Many sites block AI crawlers by default, sometimes deliberately, often by accident of a template. A blocked crawler cannot read the evidence that would get you recommended: blocking Google-Extended, for instance, withholds your content from Gemini grounding.
For anyone whose business depends on being recommended, the calculus favors access. We allow the major AI crawlers explicitly and deliberately, and checking a client’s site for accidental blocks is one of the first things our audit looks at.