Crawl budget describes the extent of crawling a search engine is willing and able to perform for a website. Google's explanation considers both crawl capacity and crawl demand.
The subject is most relevant when a site has a large or rapidly changing collection of URLs. For many smaller business websites, useful content and avoiding accidental exclusions are more immediate priorities.
Capacity versus demand
Capacity concerns the rate at which requests can be made without causing problems. Demand concerns the interest in revisiting the available material.
A fast server does not compel a search engine to request every possible URL. Equally, a valuable collection may be difficult to crawl if delivery is unreliable.
Where can unnecessary URLs arise?
Filters, parameters and alternative paths can generate many variations. Some are useful to visitors, while others repeat substantially similar information.
Investigate the actual URL patterns before restricting them. A broad rule applied without understanding the application can affect useful destinations.
Is crawl budget the same as indexing?
No. A page can be fetched without being indexed. Being indexed is also separate from ranking prominently for a search.
When a business cannot find a page in results, establish which stage is affected. Calling every absence a crawl-budget problem can lead to the wrong work.
What evidence is useful?
Review crawl reporting, delivery errors and representative URL patterns. Large sites may also use server logs to understand requests and response behaviour.
The diagnosis should identify a meaningful obstacle. A tool warning alone does not establish that commercially important content is being overlooked.
What can a business improve?
Maintain reliable delivery, coherent internal links and accurate sitemap information. Review unwanted duplication and broken routes in context.
Do not use noindex as a simplistic way to save crawl requests. The crawler normally needs to request the page to read that instruction.
How should work be prioritised?
Start with the pages and changes that matter to the business. An operational product catalogue and an old collection of parameter variations do not necessarily deserve equal attention.
A useful recommendation explains the affected pattern, supporting evidence and intended effect. Complex crawl management should solve an observed problem rather than become a default task for every site.
Further reading
For reference, see the supporting documentation. Explore related explanations of Robots.txt, XML sitemap, Technical SEO.
A practical example
Illustrative example: a large catalogue exposes many filter combinations containing similar products. The team identifies which combinations serve distinct needs and investigates how crawlers access the rest. It plans targeted changes instead of blocking the entire catalogue.
Why it matters
Crawl-budget thinking helps large sites manage discovery efficiently. Its value depends on a specific diagnosis and an understanding of the URL structure.
