Crawl budget: what it is and how to make it enough

Does crawl budget concern your site?

Probably not, if the site has a few hundred pages and new ones show up on Google within days. Crawl budget only becomes a real problem beyond a certain scale. According to Google's guide to managing crawl budget, there are three cases where it counts:

• Sites with more than one million unique pages whose content changes about once a week.
• Sites with more than 10,000 unique pages whose content changes every day.
• Sites with a substantial share of URLs reported in Search Console as “Discovered — currently not indexed”.

Google itself notes these are rough estimates, not exact thresholds. The third signal is the most useful in practice: it means Google knows about those pages but has not yet spent requests reading them. Not to be confused with “Crawled — currently not indexed”: there the page was read and set aside, and the problem is usually quality or duplication, not budget.

What crawl budget depends on

Google decides how much to crawl by weighing two things at once: how much the server takes without slowing down, and how worthwhile it is to come back to those pages. The first is a technical measure, the second an estimate of usefulness. Google crawls at most what the server bears, and in any case only as much as it finds useful. Most sites lose budget on the second while having the first in order.

Crawl capacity limit

This is the total time your server can spend holding connections open for Google: both the number of parallel connections and how long they last. It rises if the site answers quickly and reliably. It falls if response times stretch or if 5xx errors and 429 responses appear. The limit is shared by every Google crawler: if one consumes a lot of it, less is left for the others.

Rendering counts too. As Google makes clear on its page about myths and facts about crawling, the time spent rendering a page counts the same as the time spent requesting it. Pages that are lighter to process mean more pages read for the same resources.

Crawl demand

This is how much Google wants to crawl. It depends mostly on:

• Perceived inventory: how many URLs Google knows about on the site. It is the factor you control most.
• Popularity: the most linked and most searched pages get recrawled more often.
• Staleness: Google returns to a page as often as it needs to in order to catch the changes.

So two sites of the same size can be crawled at very different rates. What differs is not the number of pages, it is how many of those pages deserve a second visit.

Where crawl budget leaks

The leak is almost never one extra page. It is the same page reachable from ten different addresses, or a request spent reading nothing new.

Faceted navigation: one rule to decide by

On ecommerce sites, filters for colour, size, price and sort order are the most common cause of leakage. Every combination generates a new URL, and the crawler cannot know whether it is useful until it downloads it. There is one question to ask of each kind of filter: does this combination need to appear in search results?

1. No, it is only for users. Block those parameters in robots.txt, or handle the filters with fragments after the #, which Google does not crawl. Leave the unfiltered category page and the product pages crawlable.
2. Yes, it has search demand of its own (say “women's trail running shoes”). Make it a stable, clean URL. Always use the same filter order and avoid duplicate filters in the URL. Answer 404 when a combination has no results.

In its guide to managing the crawling of faceted navigation URLs, Google names blocking in robots.txt and fragments as the most effective methods. Canonicals and nofollow on filter links only reduce crawling over time, and nofollow works only if it is on every link pointing to that URL.

An example of rules for filter parameters:

User-agent: *
Disallow: /*?*color=
Disallow: /*?*size=
Disallow: /*?*sort=

JavaScript and rendering

Google processes pages in three phases: crawling, rendering, indexing. As the JavaScript SEO basics guide explains, pages that answer 200 are queued for rendering, unless a robots meta tag or an X-Robots-Tag header tells Google not to index them. They can sit in that queue for a few seconds or much longer, before a headless Chromium browser runs the JavaScript. Pages blocked in robots.txt are not requested at all.

For crawl budget there are three consequences:

1. If the content or the links only appear after the JavaScript runs, Google only discovers them after rendering. Discovery of new pages slows down.
2. Links have to be a elements with an href attribute. Buttons or JavaScript events without an href are not followed.
3. Blocking the JavaScript or CSS files a page needs in robots.txt stops Google seeing the page as a user sees it.

Server-side rendering or pre-rendering remain the safest choice. They make the site faster for users and crawlers alike, and not every crawler runs JavaScript.

How to optimise crawl budget, in order of return

The order matters more than the list. First stop the variants at the source, then shorten the chains, then fix the internal paths. Otherwise you are optimising paths to addresses that are about to change.

1. Stop the variants at the source. Consolidate duplicates by declaring the canonical URL, and use robots.txt to block the filters and sort orders that have no business in search.
2. Bring every redirect down to a single hop. Update internal links and the sitemap to point at the final destination.
3. Answer 404 or 410 on removed pages and get rid of soft 404s.
4. Keep the sitemap clean and current: only canonical URLs that answer 200, with lastmod updated only when the content genuinely changes.
5. Make the server fast and predictable. Cut the response time and support 304 Not Modified, so Google reuses the copy it already crawled.
6. Lighten the rendering. Fewer resources to load per page, main content and links already in the HTML.
7. Put every page that has to be crawled within a few clicks of the home page, with internal links from pages that are already visited often.

Myths worth dropping

From Google's documentation on crawling:

• “noindex saves budget.” Only partly. Google has to download the page to read the noindex, so that request is already spent. Over the long run, taking pages out of the index can free some budget indirectly. If the goal is not to have them crawled, the right tool is robots.txt.
• “I block pages in robots.txt to move budget elsewhere.” Google does not reassign the freed requests to other pages, unless the site is already hitting its capacity limit.
• “crawl-delay lets me control Googlebot.” Google ignores that rule.
• “4xx pages waste budget.” They do not: Google attempts the crawl and receives only a status code, with no content. The exception is 429, which signals rate limiting and lowers the capacity limit.
• “Compressing the sitemap increases budget.” It does not — the file still has to be downloaded.
• “Touching pages and changing the date gets them recrawled more.” Google rates quality, not age. Fake changes add nothing.
• “More crawling means better rankings.” Crawling is necessary to appear at all, but it is not a ranking factor.

How to check whether it worked

The result of a good intervention is a redistribution, not an increase: the requests concentrate on the pages that matter. Seeing that takes three sources.

Crawl stats in Search Console

They live under Settings > Crawl stats and are available only for Domain properties or at root directory level. Google describes the Crawl stats report as useful mainly for sites above a thousand pages. What to look at:

• Total crawl requests: the curve matters more than the number. A sudden drop usually has a specific cause: a new robots.txt rule, a slower server, more errors.
• Average response time: if it rises, the capacity limit falls.
• Host status: it should be green. If it is not, the detail says whether the problem is fetching robots.txt, DNS resolution or server connectivity.
• Crawl responses: most should be 200. Many 3xx means redirect chains, many 5xx means server trouble.
• Crawl purpose: “Discovery” is URLs never seen before, “Refresh” is rereads. If you are publishing new pages and discovery stays flat, Google is not finding them.
• Googlebot type: a spike in “Page resource load” points at pages that are heavy to render.

Server logs

They are the only source that shows every request, URL by URL. The method:

1. Isolate the requests that declare themselves Googlebot.
2. Verify they are genuine. The user agent can be faked. Google gives two methods in its guide to verifying crawler requests: a reverse DNS lookup of the IP, which has to resolve to googlebot.com for common crawlers or google.com for special ones, and come back to the same IP on a forward lookup; or a comparison against the IP lists Google publishes.
3. Group by section or page type: categories, products, filters, articles, service pages.
4. Measure four things: the share of non-200 requests, the share of requests on URLs you want indexed, the important pages never visited in the period, and how often the key pages are visited.
5. Compare before and after the intervention, over the same number of weeks.

The Page indexing report

Check whether “Discovered — currently not indexed” URLs are falling. It is the most direct signal that the budget is going where it should.

Migrations and multilingual sites

A migration raises crawl demand, because Google has to reread the site at the new addresses. That is the moment redirect chains and old URLs in the sitemap cost the most: every old URL had better point straight at the final destination.

On multilingual sites every language version is a URL to crawl, and versions connected with hreflang spend budget like any other. If a language sits on a subdomain, it also has a budget of its own.

Crawl budget and AI answer engines

A page that is not read cannot be cited. That holds for Google, where AI Overviews and AI Mode draw on the same index Googlebot feeds. And it holds for AI assistants that fetch web pages in real time with crawlers of their own.

Two practical consequences:

• The fixes that free budget for Googlebot hold for the other crawlers too. Fewer variants, fewer redirects and content already present in the HTML make the site more readable for all of them. Many AI crawlers do not run JavaScript, so content that only appears after rendering may not exist as far as they are concerned.
• AI crawlers consume server resources as well. They are worth watching in the logs and managing per user agent in robots.txt, blocking only what should not be read.

Crawling is the precondition of every form of organic visibility, not only of classic rankings.

Common questions