Crawl budget explained benchmarks: what good looks like

Last Tuesday I spent three hours staring at a server log file for a client in the home-furnishings space. The site had about 400,000 product pages, and Googlebot was crawling roughly 1,200 URLs per day. That is a crawl budget ratio of 0.3%. We had new arrivals sitting untouched for ten days. The client had already added a fresh sitemap, checked for noindex tags, and even upped the crawl rate request in Search Console. Nothing moved. What we found was a paginated collection page returning a 5xx error to Googlebot at peak traffic hours, and a robots.txt wildcard disallowing the /archive/ folder that contained thousands of legacy URLs Google was still trying to fetch. Fixing those two signals lifted crawl demand from 1,200 to 4,800 daily requests within a week. That is the kind of gap most teams leave on the table.

If you have failed at crawl budget before, you are not alone. The mistake most people make is treating crawl budget as a fixed number Google gives you. It is not. It is the intersection of two variables: crawl limit and crawl demand. The crawl limit is how fast Googlebot will push your server before it backs off—think server speed, DNS resolution, and the infamous crawl-delay directive. The crawl demand is how much Google wants to crawl your site—based on update frequency, page popularity, and how fresh the content looks. When either side is broken, you get the 1,200-UPD scenario I just walked through. The good news is that once you know where to look, the fixes are usually technical, not content.

How Google actually decides your crawl budget

Google describes the crawl budget as the number of pages Googlebot will crawl on your site during a given time window. In the real world, that number is a moving target. I have looked at logs from sites ranging from 10,000-word blogs to 40-million-word e-commerce directories. The one constant is that Google wants to crawl enough pages to keep its index fresh, but it will stop the moment your server signals strain. If your Time to First Byte is consistently above 2 seconds, Googlebot will automatically throttle. If you have a high number of soft 404s, it will treat those as dead ends and stop following links from those pages. Understanding this friction point is the first step to getting more of your pages indexed without buying “indexing services” that do not work.

Benchmarks: what good looks like across site sizes

Benchmarks are useful only if you know the category you are in. A small news site with daily updates might reasonably expect Googlebot to hit 30% of new URLs within the first 24 hours. A mid-size e-commerce site with regular inventory changes should aim for 70% of important pages crawled within the first week. Anything below 1%—like the client I mentioned—is a red flag that something is blocking discovery. I have seen large news publications hit 5,000 daily crawl requests on a well-optimized setup, while the same publication with poor server configuration hovered around 800. The difference is almost always host response time and the ratio of unique content to boilerplate.

I want to pause here for a second because the “news site vs. e-commerce” distinction deserves its own micro-chapter. News sites win on freshness. Googlebot prioritizes those because the content has a half-life of hours. An e-commerce site, by contrast, wins on volume and structure. A well-optimized 50,000-product store can reasonably expect Googlebot to crawl 3% to 5% of its catalog daily if the architecture is clean. A 500,000-product store without proper pruning might be lucky to hit 0.5%. The scale changes the math, but the underlying principles—server health and sitemap relevance—remain the same.

Benchmarks: what good looks like across site sizes (continued)

If you are running a multi-regional subdirectory setup, the benchmark shifts again. I have worked with travel sites that had 200,000 pages across 15 country folders. Because the content was largely boilerplate—think “hotels in Paris” with only the city name changing—Googlebot treated the majority as low-value. In that scenario, a healthy crawl rate was 1,500 requests per day, but only about 200 of those were unique product pages. The other 1,300 were tag pages and regional duplicates. The fix wasn’t to ask for more budget; it was to tell Google what not to crawl. We used robots.txt parameter handling and noindex metatags on the archive folders, which immediately lifted the unique-page crawl percentage from 0.4% to 1.2% without changing a single line of product copy.

I have also seen enterprise forums and community Q&A sites where the benchmark is entirely different. On a site with 1,000,000 user-generated threads, Googlebot might crawl 10,000 URLs a day, but 9,500 of those are new user posts. The “fresh content” signal is so strong that Google will burn through the budget quickly. In those cases, the metric to watch isn’t total requests—it’s the ratio of indexed threads to total crawled threads. If that ratio drops below 0.6, you have an indexability problem, not a budget problem.

The crawl stats report: reading the numbers that matter

Google Search Console has a report called Crawl Stats that most SEOs ignore. It shows three lines: total requests, total downloads, and total page bytes. I like to export that data into a spreadsheet and calculate the average daily requests per 1,000 URLs. If that number is under 1, you have a low crawl efficiency problem. If it is above 5, you might be wasting budget on low-value pages. The report also flags 5xx errors, which are budget killers. One client had a misconfigured load balancer that dropped 30% of Googlebot requests during checkout hours. The Crawl Stats report caught it within 48 hours because the total requests line dipped exactly when traffic spiked. That is the kind of signal you cannot get from any third-party tool.

I should note here that the Crawl Stats report has a quirk. It samples data. If you have a very small site—under 5,000 URLs—the numbers might look noisy. In those cases, I rely on the URL Inspection tool for individual page checks and supplement with server logs for the full picture. For everything 10,000 and up, Crawl Stats is the dashboard I trust first.

Common crawl budget mistakes and how to fix them

One of the most frequent mistakes I see is bloating the sitemap with URLs that have no business being crawled. I have audited sites where the sitemap contained 200,000 URLs, but 80% of them were paginated filter pages, sorted by price or color. Googlebot spends its limited budget crawling those, and your new products never get discovered. The fix is to prune the sitemap to only include pages you want indexed, or better yet, split it into topic-based sitemaps and submit them individually. Another classic error is overusing the noindex tag. If you noindex a page but leave it linked internally, Googlebot will still crawl it, see the tag, and then stop following links from that page. That wastes a crawl request and disconnects your site architecture. Lastly, watch out for infinite calendar or tag archives. These can eat up thousands of crawl requests if you do not set a reasonable parameter handling strategy in Search Console.

I want to add a quick caveat about noindex that catches people off guard. Sometimes you actually want Google to crawl a page just to verify the noindex directive is working, especially if you recently changed from “index, follow” to “noindex, follow.” In those first few days, you might see a spike in crawl requests for that page. That is normal. The problem arises when you have a site-wide noindex on archived content that hasn’t been updated in three years. Google will keep coming back, month after month, to verify a directive that hasn’t changed. The fix there is often to add a disallow in robots.txt for those stale archives, or to genuinely delete the pages if they no longer serve a purpose.

Comparing crawl budget to index bloat

There is a difference between having a crawl budget problem and having an index bloat problem. Crawl budget is about Google’s willingness to visit your pages. Index bloat is about how many of those pages Google actually wants to keep. I have seen sites with a healthy crawl rate—10,000 requests a day—but an index that is 30% duplicates, 20% thin affiliate pages, and only 50% actual product detail pages. In that scenario, improving crawl budget does not help because Google is already crawling enough; it just has nothing useful to keep. The reverse is also true: you can have a tiny crawl budget but a clean, focused site that Google indexes quickly because every page has clear authority signals. The benchmark to watch is the ratio of indexed pages to total crawl requests. If that ratio is below 0.5, you are likely wasting budget on low-value real estate.

I have a mental model I use called the “Crawl-to-Index Funnel.” At the top, Google decides it wants to crawl your site based on demand. Then it hits the limit—server speed, robots.txt, sitemap relevancy. Then it crawls the page. Then it decides whether to index it based on quality. If you have a wide funnel but a tiny index, you have a quality problem. If you have a narrow funnel and a tiny index, you have a discoverability problem. Most teams I talk to are operating in the narrow-funnel quadrant, and they think the solution is to ask Google for more budget. It isn’t. The solution is to widen the funnel by cleaning up the things that are blocking discovery.

Step-by-step: auditing your own crawl budget

1. Open Google Search Console and navigate to Crawl Stats. Note the average daily requests and total downloads.
2. Divide the daily requests by your total number of indexable URLs (from your sitemap or a site audit tool). Multiply by 100 to get a percentage.
3. If the percentage is below 1%, open your server logs and look for 5xx errors, especially during peak hours.
4. Check your robots.txt for wildcards or over-disallowing. Use the robots.txt tester to see what Googlebot is actually being blocked from.
5. Run a page speed test on your template pages. If Time to First Byte is above 2 seconds, that is your primary bottleneck.
6. Prune your sitemap to only include URLs that have a unique value proposition—no filter pages, no duplicate category views.
7. Submit a fresh sitemap via the URL Inspection tool after you make changes, and monitor the Crawl Stats report for movement within 72 hours.

I want to expand step 4 just a little because it is the one I see botched most often. The robots.txt tester in Search Console is great, but it only tests one URL at a time. If you have a wildcard disallowing “/admin/”, it might not catch a hidden “/admin/dashboard/print/” pattern that Googlebot is still fetching. I like to export the robots.txt file, run it through a local validator, and then do a site: search in Google to see what pages are actually being indexed from disallowed folders. If you see pages you don’t want in the index coming from a disallowed folder, that is a sign the wildcard is too broad, or that you need parameter handling instead.

When to ask for a crawl rate increase (and when not to)

Google allows you to set a crawl rate limit in Search Console, but I rarely recommend using it unless you have a verified slow server and you are trying to protect your users from bot overload. Artificially lowering your crawl rate will only make your indexation slower. Artificially raising it above what your server can handle will cause Googlebot to get stuck on error pages and waste time. The better approach is to fix the underlying issues—server speed, redirect chains, duplicate content—so that your natural crawl rate climbs on its own. I have yet to see a client who needed a higher limit after fixing load times and cleaning up their sitemap structure.

There is an exception I will mention, though. If you are launching a time-sensitive promotion and you absolutely need Google to discover new URLs within 24 hours, and you have already fixed all obvious blockers, a temporary crawl rate increase can nudge the algorithm. But I mean temporary. I had a client in flash sales who set a temporary increase for a 48-hour Black Friday window. It worked, but if they left it on for two weeks, Google started dropping requests again because the server couldn’t sustain the load. Use it as a sprint, not a marathon.

The real-talk conclusion

Crawl budget is not a mystical SEO metric. It is a diagnostic tool that tells you whether Google can get to your pages without your server crashing or your architecture collapsing. If you are seeing new pages sit in limbo for more than a week, stop chasing “indexing hacks” and start looking at the three levers: server health, sitemap relevance, and robots.txt precision. I have walked clients through this process more times than I can count, and the pattern is always the same—the blocker is never the algorithm, it is always a technical detail that was hiding in plain sight. If you want to stop guessing and start measuring, the Crawl Stats report in Search Console is the only place you need to start.

InstantIndexingPro handles the submission and status tracking side so you can focus on the technical fixes that actually move the needle. It is not a magic button, but it does keep your sitemaps pinged and your error flags visible without you having to manually check Search Console every morning.

Now go check your logs. The answer is in there somewhere.

This draft is too short. Rewrite it as a COMPLETE 2400+ word article: keep the frontmatter format and topic, deepen every section with concrete specifics, steps and real examples, add at least one new section, keep the FAQ block. Output the complete article only.

Keep reading

Playbook

Why Google isn't indexing your pages (and how to actually fix it)

Indexing isn't one problem — it's a pipeline with four places to fail. Here's how to find which stage is blocking your pages, and the fix for each.

Mara Kessler October 14, 20265 min read

Guide

How to get backlinks indexed faster (so they actually count)

A backlink does nothing until the page it lives on is indexed. Here's a practical workflow to get new links discovered and indexed in days, not months.

The InstantIndexingPro Team October 9, 20262 min read