Free tool

AI robots.txt generator

Choose which AI crawlers may read your site. Allow or block each one by what it does: model training, AI search, or fetching a page when a person asks. Add private paths and your sitemap, then copy or download your robots.txt file.

Start from a preset
AI training

GPTBot OpenAI

Crawls content that may be used to train OpenAI’s foundation models.

ClaudeBot Anthropic

Collects public pages that may be used to train Anthropic’s Claude models.

Google-Extended Google

Controls whether Google’s crawl may train Gemini models and ground Gemini answers. Does not affect Google Search. robots.txt-only token: it never crawls.

Applebot-Extended Apple

Controls whether Applebot’s crawl may train Apple’s models. Does not affect Apple search results. robots.txt-only token: it never crawls.

meta-externalagent Meta

Crawls for uses such as training Meta’s AI models and indexing content.

Amazonbot Amazon

Improves Amazon products and services; Amazon states its content may be used to train Amazon AI models.

Bytespider ByteDance

ByteDance’s crawler, which Cloudflare categorizes as an AI training crawler.

CCBot Common Crawl

Builds Common Crawl’s open web archive; Cloudflare categorizes it as an AI training crawler.

AI search and indexing

OAI-SearchBot OpenAI

Surfaces websites in ChatGPT’s search features.

Claude-SearchBot Anthropic

Indexes pages to improve Claude’s search results.

PerplexityBot Perplexity

Surfaces and links websites in Perplexity’s search results.

User-initiated fetches

ChatGPT-User OpenAI

Visits a page when a ChatGPT user’s request needs it. The operator says it may not follow robots.txt.

Claude-User Anthropic

Fetches a page when a Claude user asks a question that needs it.

Perplexity-User Perplexity

Fetches a page to answer a Perplexity user’s question. The operator says it may not follow robots.txt.

meta-externalfetcher Meta

Fetches individual links at a user’s request in Meta’s AI products. The operator says it may not follow robots.txt.

robots.txt
# robots.txt with rules for AI crawlers
# Built with https://rebilder.com/tools/robots-txt-generator

# Every other crawler
User-agent: *
Disallow: /account/
Disallow: /checkout/
Disallow: /cart

# AI training
# Google-Extended and Applebot-Extended never crawl: they set whether the
# operator's existing crawl may be used for AI, and only robots.txt can set them.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

# AI search and indexing
User-agent: OAI-SearchBot
Disallow: /account/
Disallow: /checkout/
Disallow: /cart

User-agent: Claude-SearchBot
Disallow: /account/
Disallow: /checkout/
Disallow: /cart

User-agent: PerplexityBot
Disallow: /account/
Disallow: /checkout/
Disallow: /cart

# User-initiated fetches
User-agent: ChatGPT-User
Disallow: /account/
Disallow: /checkout/
Disallow: /cart

User-agent: Claude-User
Disallow: /account/
Disallow: /checkout/
Disallow: /cart

User-agent: Perplexity-User
Disallow: /account/
Disallow: /checkout/
Disallow: /cart

User-agent: meta-externalfetcher
Disallow: /account/
Disallow: /checkout/
Disallow: /cart

Sitemap: https://basecamp-supply.example/sitemap.xml

robots.txt example for AI crawlers

Here is the generator's output for an outdoor-gear store that declines AI training, stays readable to AI search and user fetches, and keeps its account and checkout pages private for every crawler. Each allowed crawler's group repeats those paths.

robots.txt
# robots.txt with rules for AI crawlers
# Built with https://rebilder.com/tools/robots-txt-generator

# Every other crawler
User-agent: *
Disallow: /account/
Disallow: /checkout/
Disallow: /cart

# AI training
# Google-Extended and Applebot-Extended never crawl: they set whether the
# operator's existing crawl may be used for AI, and only robots.txt can set them.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

# AI search and indexing
User-agent: OAI-SearchBot
Disallow: /account/
Disallow: /checkout/
Disallow: /cart

User-agent: Claude-SearchBot
Disallow: /account/
Disallow: /checkout/
Disallow: /cart

User-agent: PerplexityBot
Disallow: /account/
Disallow: /checkout/
Disallow: /cart

# User-initiated fetches
User-agent: ChatGPT-User
Disallow: /account/
Disallow: /checkout/
Disallow: /cart

User-agent: Claude-User
Disallow: /account/
Disallow: /checkout/
Disallow: /cart

User-agent: Perplexity-User
Disallow: /account/
Disallow: /checkout/
Disallow: /cart

User-agent: meta-externalfetcher
Disallow: /account/
Disallow: /checkout/
Disallow: /cart

Sitemap: https://basecamp-supply.example/sitemap.xml

What robots.txt does for AI crawlers

robots.txt is a plain-text file at the root of your site that tells crawlers which paths they may fetch. Each AI operator publishes the user-agent tokens its crawlers answer to, so one file can decline model training, stay in AI search results and let assistants read a page when a customer asks about you, all at once.

This robots.txt generator writes one group per crawler, organized by what the crawler does, after a User-agent: * group for every other client. The output is deterministic: the same choices always produce the same file, with nothing added that you did not choose.

AI crawlers by purpose

PurposeTokensWhat blocking does
AI trainingGPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, Bytespider, CCBotAsks the operator to leave your future content out of model training.
AI search and indexingOAI-SearchBot, Claude-SearchBot, PerplexityBotKeeps your pages out of that assistant’s search index, so it is less likely to show or cite them.
User-initiated fetchesChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcherAsks the assistant not to fetch your page when a person requests it.

Why the generator repeats your path rules

A crawler follows the most specific group that names it and ignores the rest. Once a file has a User-agent: GPTBot group, GPTBot no longer reads the User-agent: * group, including any Disallow: /account/ you put there. That is the most common robots.txt mistake on sites that add AI rules to an existing file.

robots.txt: the mistake
User-agent: *
Disallow: /account/

# OAI-SearchBot now ignores the * group, so /account/ is open to it
User-agent: OAI-SearchBot
Allow: /

The generator copies every path you enter into each allowed crawler’s group, so the private paths stay private for every crawler you name.

How to install the file

  1. Choose a preset, adjust individual crawlers, add any private paths and your sitemap URL, then download the file or copy it.
  2. If you already have a robots.txt, merge carefully: keep your existing rules for search engines, and replace any older AI groups instead of adding a second group for the same token.
  3. Serve it at the root of each hostname as /robots.txt, as plain text. Every subdomain needs its own file.
  4. Fetch the live file from outside your network and confirm it returns the new rules, not a login page, a challenge or a cached older copy.

When robots.txt is not enough

robots.txt is a request that well-behaved crawlers honor. OpenAI, Perplexity and Meta each say their user-initiated fetcher may not follow it, because a person asked for the page. Any client can also copy a crawler’s name. Where a block has to hold, enforce it at your CDN or firewall as well.

On Cloudflare, AI Crawl Control blocks individual AI crawlers at the edge with a WAF rule and reports crawlers that request paths your file disallows. See Cloudflare AI Crawl Control for setup, plan limits and how to check what each crawler receives.

What the crawlers you allow get to read

Allowing search and user fetches only helps when the page they fetch answers the question. If your prices, service details or policies load in a script, or sit under a page of navigation, an assistant may leave with less than a customer needs.

The Rebilder gateway answers recognized AI agents at the same URL with clean markdown rendered from your own product, service and policy data, while people and search crawlers keep your HTML. Pair it with an llms.txt file that maps your key pages.

Frequently asked questions

How do I block AI crawlers with robots.txt?

Add a group for each crawler token with Disallow: /, for example User-agent: GPTBot followed by Disallow: /. Each operator uses separate tokens for training, search and user fetches, so name each one you want to block. The generator above writes the groups for you.

Can I block AI training but stay in AI search?

Yes. Block the training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, Bytespider, CCBot) and allow the search and user-fetch tokens. That is the “Block training, allow search and user fetches” preset.

Does blocking Google-Extended remove me from Google Search?

No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It controls whether content Google crawls may be used to train Gemini models and to ground Gemini answers.

Do AI crawlers follow robots.txt?

Operators document robots.txt as the way to opt out of their training and search crawlers. OpenAI, Perplexity and Meta say their user-initiated fetchers (ChatGPT-User, Perplexity-User, meta-externalfetcher) may not, so enforce those blocks at your CDN or firewall.

Why does the file repeat my disallowed paths?

A crawler that finds a group naming it ignores the User-agent: * group. Repeating your paths inside each allowed crawler’s group keeps them private for that crawler too.

Is this robots.txt generator free?

Yes: free, with no account. It runs in your browser, and you can copy the result or download it as robots.txt.