Glossary · Technical
ClaudeBot
Anthropic's web crawler, used to collect content for training its Claude models.
What it does
ClaudeBot crawls publicly accessible pages following standard robots.txt conventions, similar to how GPTBot and other AI crawlers operate — controllable per-site via a specific user-agent directive.
Relevance to visibility
As with other AI crawlers, allowing ClaudeBot means new content can inform how future versions of Claude describe a brand, while blocking it has no retroactive effect on the model's existing knowledge.
See more terms in the full glossary, or read the AI Visibility 101 guide for the full picture.
Related