Glossary · Technical
Crawl Budget
The number of pages a crawler will visit on a given site within a given timeframe.
Why it's limited
Crawlers — search engine or AI — don't have unlimited time or resources to spend on any one site, so they allocate a budget based on signals like site size, update frequency and perceived importance.
Why it matters at scale
For very large sites, an inefficient structure can mean a crawler never reaches lower-priority pages within its budget — keeping important content easy to discover (clear navigation, sitemaps, minimal duplicate content) helps make the most of a limited crawl budget.
See more terms in the full glossary, or read the AI Visibility 101 guide for the full picture.