Free tool
AI robots.txt generator
Choose which AI crawlers may read your site. Allow or block each one by what it does: model training, AI search, or fetching a page when a person asks. Add private paths and your sitemap, then copy or download your robots.txt file.
# robots.txt with rules for AI crawlers # Built with https://rebilder.com/tools/robots-txt-generator # Every other crawler User-agent: * Disallow: /account/ Disallow: /checkout/ Disallow: /cart # AI training # Google-Extended and Applebot-Extended never crawl: they set whether the # operator's existing crawl may be used for AI, and only robots.txt can set them. User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: meta-externalagent Disallow: / User-agent: Amazonbot Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / # AI search and indexing User-agent: OAI-SearchBot Disallow: /account/ Disallow: /checkout/ Disallow: /cart User-agent: Claude-SearchBot Disallow: /account/ Disallow: /checkout/ Disallow: /cart User-agent: PerplexityBot Disallow: /account/ Disallow: /checkout/ Disallow: /cart # User-initiated fetches User-agent: ChatGPT-User Disallow: /account/ Disallow: /checkout/ Disallow: /cart User-agent: Claude-User Disallow: /account/ Disallow: /checkout/ Disallow: /cart User-agent: Perplexity-User Disallow: /account/ Disallow: /checkout/ Disallow: /cart User-agent: meta-externalfetcher Disallow: /account/ Disallow: /checkout/ Disallow: /cart Sitemap: https://basecamp-supply.example/sitemap.xml
robots.txt example for AI crawlers
Here is the generator's output for an outdoor-gear store that declines AI training, stays readable to AI search and user fetches, and keeps its account and checkout pages private for every crawler. Each allowed crawler's group repeats those paths.
# robots.txt with rules for AI crawlers
# Built with https://rebilder.com/tools/robots-txt-generator
# Every other crawler
User-agent: *
Disallow: /account/
Disallow: /checkout/
Disallow: /cart
# AI training
# Google-Extended and Applebot-Extended never crawl: they set whether the
# operator's existing crawl may be used for AI, and only robots.txt can set them.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
# AI search and indexing
User-agent: OAI-SearchBot
Disallow: /account/
Disallow: /checkout/
Disallow: /cart
User-agent: Claude-SearchBot
Disallow: /account/
Disallow: /checkout/
Disallow: /cart
User-agent: PerplexityBot
Disallow: /account/
Disallow: /checkout/
Disallow: /cart
# User-initiated fetches
User-agent: ChatGPT-User
Disallow: /account/
Disallow: /checkout/
Disallow: /cart
User-agent: Claude-User
Disallow: /account/
Disallow: /checkout/
Disallow: /cart
User-agent: Perplexity-User
Disallow: /account/
Disallow: /checkout/
Disallow: /cart
User-agent: meta-externalfetcher
Disallow: /account/
Disallow: /checkout/
Disallow: /cart
Sitemap: https://basecamp-supply.example/sitemap.xml
What robots.txt does for AI crawlers
robots.txt is a plain-text file at the root of your site that tells crawlers which paths they may fetch. Each AI operator publishes the user-agent tokens its crawlers answer to, so one file can decline model training, stay in AI search results and let assistants read a page when a customer asks about you, all at once.
This robots.txt generator writes one group per crawler, organized by what the crawler does, after a User-agent: * group for every other client. The output is deterministic: the same choices always produce the same file, with nothing added that you did not choose.
AI crawlers by purpose
| Purpose | Tokens | What blocking does |
|---|---|---|
| AI training | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, Bytespider, CCBot | Asks the operator to leave your future content out of model training. |
| AI search and indexing | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Keeps your pages out of that assistant’s search index, so it is less likely to show or cite them. |
| User-initiated fetches | ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher | Asks the assistant not to fetch your page when a person requests it. |
Why the generator repeats your path rules
A crawler follows the most specific group that names it and ignores the rest. Once a file has a User-agent: GPTBot group, GPTBot no longer reads the User-agent: * group, including any Disallow: /account/ you put there. That is the most common robots.txt mistake on sites that add AI rules to an existing file.
User-agent: *
Disallow: /account/
# OAI-SearchBot now ignores the * group, so /account/ is open to it
User-agent: OAI-SearchBot
Allow: /The generator copies every path you enter into each allowed crawler’s group, so the private paths stay private for every crawler you name.
How to install the file
- Choose a preset, adjust individual crawlers, add any private paths and your sitemap URL, then download the file or copy it.
- If you already have a
robots.txt, merge carefully: keep your existing rules for search engines, and replace any older AI groups instead of adding a second group for the same token. - Serve it at the root of each hostname as
/robots.txt, as plain text. Every subdomain needs its own file. - Fetch the live file from outside your network and confirm it returns the new rules, not a login page, a challenge or a cached older copy.
When robots.txt is not enough
robots.txt is a request that well-behaved crawlers honor. OpenAI, Perplexity and Meta each say their user-initiated fetcher may not follow it, because a person asked for the page. Any client can also copy a crawler’s name. Where a block has to hold, enforce it at your CDN or firewall as well.
On Cloudflare, AI Crawl Control blocks individual AI crawlers at the edge with a WAF rule and reports crawlers that request paths your file disallows. See Cloudflare AI Crawl Control for setup, plan limits and how to check what each crawler receives.
What the crawlers you allow get to read
Allowing search and user fetches only helps when the page they fetch answers the question. If your prices, service details or policies load in a script, or sit under a page of navigation, an assistant may leave with less than a customer needs.
The Rebilder gateway answers recognized AI agents at the same URL with clean markdown rendered from your own product, service and policy data, while people and search crawlers keep your HTML. Pair it with an llms.txt file that maps your key pages.
Frequently asked questions
How do I block AI crawlers with robots.txt?
Add a group for each crawler token with Disallow: /, for example User-agent: GPTBot followed by Disallow: /. Each operator uses separate tokens for training, search and user fetches, so name each one you want to block. The generator above writes the groups for you.
Can I block AI training but stay in AI search?
Yes. Block the training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, Bytespider, CCBot) and allow the search and user-fetch tokens. That is the “Block training, allow search and user fetches” preset.
Does blocking Google-Extended remove me from Google Search?
No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It controls whether content Google crawls may be used to train Gemini models and to ground Gemini answers.
Do AI crawlers follow robots.txt?
Operators document robots.txt as the way to opt out of their training and search crawlers. OpenAI, Perplexity and Meta say their user-initiated fetchers (ChatGPT-User, Perplexity-User, meta-externalfetcher) may not, so enforce those blocks at your CDN or firewall.
Why does the file repeat my disallowed paths?
A crawler that finds a group naming it ignores the User-agent: * group. Repeating your paths inside each allowed crawler’s group keeps them private for that crawler too.
Is this robots.txt generator free?
Yes: free, with no account. It runs in your browser, and you can copy the result or download it as robots.txt.