Glossary · Technical
AI Crawler
An automated bot operated by an AI company that visits and indexes web content, separate from traditional search engine crawlers.
Why they're tracked separately
AI crawlers — like GPTBot, ClaudeBot and PerplexityBot — collect content specifically for AI training or retrieval purposes, distinct from the crawlers search engines use for their own indexes. A site can allow one and block another.
Controlling access
Site owners control AI crawler access the same way they control traditional search crawlers — through robots.txt directives naming each specific bot by its user-agent.
See more terms in the full glossary, or read the AI Visibility 101 guide for the full picture.
Related