What Is Crawl Budget?
Also known as Crawl rate, Crawl capacity, Crawl demand
Crawl budget is the number of URLs a search engine like Google is willing and able to crawl on a website within a given period. Google describes it as a combination of crawl capacity, how much crawling a server can handle, and crawl demand, how much Google wants to crawl. It mainly matters for very large or fast-changing sites.
Crawl Budget explained
Crawl capacity rises when a server responds quickly and reliably and falls when it slows down or returns errors. Crawl demand depends on how popular and fresh pages are and how many URLs Google knows about. Together they decide how many pages Googlebot fetches. If important pages are not crawled, they cannot be indexed or updated in results.
Google's own guidance says most sites do not need to worry. Sites with a few thousand pages are usually crawled fine. Crawl budget becomes a real concern for very large sites, such as big ecommerce catalogs, marketplaces or programmatic SEO projects with hundreds of thousands of URLs, or sites whose content changes daily, where new pages can wait a long time to be discovered.
The main problem is waste: Googlebot spending its time on URLs that do not matter. Common sources are faceted filters and sort parameters, internal search results, calendar pages, session IDs, redirect chains, soft 404s and duplicate content. Blocking or consolidating these, fixing server speed and keeping sitemaps clean lets crawlers focus on pages that should rank.
Crawl Budget example
A B2B parts marketplace has around 80,000 product pages but filter combinations create millions of crawlable parameter URLs. Search Console's Crawl Stats report shows Googlebot spending most requests on filtered listing pages, while new products take weeks to get indexed. The team blocks low-value filter parameters in robots.txt, adds canonical tags to remaining variants and lists only canonical product URLs in sitemaps. New products start getting indexed much faster.
User-agent: *
Disallow: /search/
Disallow: /*?sort=
Disallow: /*?sessionid=
Disallow: /*&filter=
Sitemap: https://www.example.com/sitemap.xmlHow to improve crawl budget
Check whether it matters
Review site size and Search Console's Page indexing report. If important pages are indexed promptly, crawl budget is probably not your bottleneck.
Read the Crawl Stats report
In Search Console settings, review total requests, response codes, file types and average response time to spot waste and server problems.
Block low-value URLs
Use robots.txt to stop crawling of internal search, endless filters, sort parameters and other URLs you never want ranked.
Consolidate duplicates
Use canonical tags and 301 redirects to fold duplicate URLs into one version, and fix redirect chains.
Speed up the server
Faster, error-free responses raise crawl capacity. Fix 5xx errors and slow responses on large templates first.
Keep sitemaps clean
List only canonical, indexable URLs with accurate lastmod dates so Google can prioritize fresh pages.
Common mistakes
- Worrying about crawl budget on a small site where it is almost never the real issue.
- Using noindex to save crawl budget, when Google still has to crawl a page to see the tag.
- Blocking pages in robots.txt that need to be crawled for their noindex or canonical tags to be seen.
- Letting faceted navigation generate endless crawlable parameter combinations that Googlebot keeps fetching.
- Relying on the crawl-delay directive, which Googlebot does not support.
Crawl Budget FAQ
Usually not. Google's guidance says crawl budget mostly matters for very large sites or sites with content that changes very frequently. If your site has a few thousand pages and new pages get indexed within days, focus on content quality and internal linking instead.
Use the Crawl Stats report in Google Search Console, found under Settings. It shows total crawl requests, download size, average response time, response codes and the purpose of crawls over recent months. Server log analysis gives an even more detailed view of exactly which URLs Googlebot requests.
Not really. Google has to crawl a page to see the noindex tag, so those pages still use crawl requests, though Google may crawl them less often over time. To stop crawling entirely, use robots.txt. Just remember robots.txt blocks crawling, not indexing.
Publish and update useful content regularly, earn links, keep strong internal links to new pages, maintain fast and reliable server responses and submit accurate XML sitemaps. For individual important pages, the URL Inspection tool in Search Console lets you request indexing.