AI crawler access: when robots.txt hides you from answer engines

Separate ranking crawlers from answer crawlers

Googlebot still needs your pages if you want them in Search. GPTBot, ClaudeBot, PerplexityBot, and Google-Extended are a different decision: they fetch pages that answer engines may quote. A privacy template that disallows every unfamiliar user-agent does not tell those jobs apart.

Teams then see a confusing split. Classic rankings hold while citations vanish, or a guide is quoted from an old copy because the crawler that would have refreshed it was turned away. Search visibility and generative visibility fail for different reasons, and a single Disallow line can cause the second one.

• Keep Googlebot allowed on every URL you want in Search.
• Name the AI crawlers you intend to refuse instead of using a blanket disallow.
• Treat Google-Extended as a generative control, not as a substitute for Googlebot.

Allow the pages you want quoted

If a guide, service page, or methodology note is meant to be a source, the fetching bot needs a normal 200 response and the same HTML a reader would get. Blocking /guides, then hoping for GEO citations, is a contradiction. An llms.txt file can point at those URLs, but it cannot replace access.

Keep disallows for admin paths, previews, and infinite filter spaces. Leave the content host fetchable for the bots you want as readers. A page that is citation-ready on screen and forbidden in robots.txt is invisible to the system you wrote it for.

Check the live file, not the policy slide

Fetch robots.txt on the production host and test a real guide URL as GPTBot and as Google-Extended. Staging rules are a frequent cause: a prelaunch Disallow: / ships with the release, and every answer crawler stops at the front door. The file in the repository is only useful if it matches the file on the live host.

After you open access, watch server logs for those user-agents and whether the URLs they fetch match your canonical pages. Access is the first condition. A clear, bounded answer is still what gets reused. If the logs stay empty, the block is still in place or the bot has not returned yet.

Should a site block GPTBot?

Block GPTBot only when you do not want that system to fetch your pages for answers. Blocking it does not, by itself, remove you from Google Search, but it does remove one route into AI answers.

Does Google-Extended control classic rankings?

No. Google-Extended is a control for some of Google's generative features. Leave Googlebot able to crawl every URL you want in Search, and decide on Google-Extended separately.