Crawl budget waste: when bots spend the day on the wrong URLs

See what is actually being fetched

Search Console’s crawl stats and a sample of server logs show which paths consume fetches. Faceted URLs, calendar pages, session parameters, and infinite scroll archives often outnumber the pages you care about.

A small brochure site rarely has a crawl-budget emergency. Treat the issue as real when important templates are slow to be recrawled after updates.

Cut the paths that do not deserve a fetch

Disallow or noindex parameter combinations that do not create a unique document. Canonicalise duplicates. Return a proper status for gone pages instead of a soft empty shell.

Do not block resources that rendering needs, such as CSS and JavaScript. Saving a few fetches there can hide the page from a renderer.

• Block useless parameter spaces in robots.txt when they should not be crawled.
• Keep the XML sitemap limited to canonical, indexable URLs.
• Fix redirect chains so one hop does not become three fetches.

Spend the recovered crawl on updates

After the noise drops, resubmit the sitemap and watch whether updated service pages and guides are fetched sooner. The metric that matters is recrawl time for URLs that change, not a larger total crawl count.

Log the patterns you blocked. A redesign that reintroduces faceted links will spend the budget again within a release.

Do small websites need to worry about crawl budget?

Usually not. If the site has a few hundred clean URLs, fix indexation and internal links first. Crawl budget becomes practical when filters, parameters, or archives create thousands of low-value URLs.

Should robots.txt be used to hide low-quality pages?

Use robots.txt to stop crawling of infinite or useless URL spaces. Use noindex when the URL can still be crawled but should not appear in search. A blocked URL cannot be noindexed reliably.