Glossary · Content & Optimization
AI Crawler Allowlisting
Deliberately permitting specific AI bots, like GPTBot, ClaudeBot and PerplexityBot, to access a site via robots.txt rather than blocking them by default.
Why defaults matter
Many CDNs and security tools now block unrecognized bots by default, which can silently prevent AI crawlers from ever reaching a site's content, even when the owner never intended to exclude them.
Checking the whole stack
Confirming allowlisting means checking both robots.txt and any firewall or bot-management layer sitting in front of it. A permissive robots.txt can still be overridden by infrastructure blocking the same user-agents further upstream.
See more terms in the full glossary, or read the AI Visibility 101 guide for the full picture.
Related