Crawl budget is the number of page fetches a crawler grants a website within a given period. It depends on the size of the website, its response speed and its importance. Crawl management is the work of making sure this budget goes to the content that really counts — and not to filter pages, parameter URLs and duplicates.
How does the crawl budget work?
A crawler discovers URLs through links and sitemaps (URL Discovery) and works through them in an order it determines itself. Every URL costs one fetch. If the server responds slowly (TTFB), the number of fetches per unit of time drops. If the website contains many URLs without value of their own — sort orders, session parameters, internal search results — the budget is spread across them, and the important pages are visited less often or later.
For LLM crawlers this applies even more strictly: they return less often than Googlebot and do not retry aborted fetches.
Why does the crawl budget matter for AI visibility?
Because content a crawler does not reach makes it neither into training nor into a live answer. The crawl budget is part of the chain that precedes every ranking: URL Discovery, crawl management, index management, linking, content quality. If one link fails, the rest happens without the page.
What does this mean for your website?
Keep the XML sitemap current and restrict it to indexable URLs. Exclude parameter, filter and duplicate URLs from crawling. Mask links that crawlers should not follow. Check in the log files which URLs AI bots actually fetch — if filter pages appear there instead of service pages, the budget is misallocated.