Robots.txt blocks that hide the pages you meant to rank

Read the file as a crawler would

Check the live robots.txt against every important template: guides, services, category pages, and the files needed to render them. A disallow of /guides or a wildcard that catches query strings on content URLs is enough to stall a site.

Staging rules are a frequent cause. A production deploy that keeps Disallow: / from a prelaunch file blocks the entire host.

Block spaces, not documents

Robots.txt is the right tool for infinite URL spaces: internal search, useless filters, and admin paths. It is the wrong tool for taking a known page out of the index, because a blocked URL may remain indexed without an update.

Allow CSS and JavaScript that the page needs. Blocking them saves nothing meaningful and can prevent rendering.

• Test the robots.txt against a live content URL.
• Remove leftover prelaunch disallows.
• Do not use robots.txt as a substitute for noindex.

Confirm the next crawl can fetch the page

After the file is fixed, use URL inspection to confirm the URL is allowed. Then watch whether the page is recrawled and whether the latest title appears in results.

Keep robots.txt in version control next to the rest of the site so a deploy review can see the diff.

If I disallow a URL, will it leave the index?

Not reliably. A blocked URL cannot be recrawled, so an old copy can remain indexed. Use noindex, or return 404, when the goal is removal.

Should sitemap URLs be blocked in robots.txt?

No. URLs you list in the sitemap should be crawlable and indexable. A conflict between the two files wastes both.