Free tool

robots.txt checker and validator

Paste your robots.txt or fetch it from your site, then test any URL. See whether Googlebot, Bingbot and every major AI crawler may fetch it, which group and rule decided, and which lines crawlers will misread or ignore.

18 of 19 crawlers blocked from /account/orders · 3 groups · 0 errors, 2 warnings

Results for /account/orders
CrawlerResultGroup usedDeciding rule
Search engines
Googlebot GoogleBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
bingbot MicrosoftBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
Applebot AppleBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
DuckDuckBot DuckDuckGoBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
AI training
GPTBot OpenAIBlockedUser-agent: GPTBot (line 7)Disallow: / (line 9)
ClaudeBot AnthropicBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
Google-Extended GoogleControl token: never crawls, so robots.txt is the only place to set it.BlockedUser-agent: Google-Extended (line 7)Disallow: / (line 9)
Applebot-Extended AppleControl token: never crawls, so robots.txt is the only place to set it.BlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
meta-externalagent MetaBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
Amazonbot AmazonBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
Bytespider ByteDanceBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
CCBot Common CrawlBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
AI search and indexing
OAI-SearchBot OpenAIAllowedUser-agent: OAI-SearchBot (line 11)Allow: / (line 12)
Claude-SearchBot AnthropicBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
PerplexityBot PerplexityBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
User-initiated fetches
ChatGPT-User OpenAIIts operator says this fetcher may not follow robots.txt.BlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
Claude-User AnthropicBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
Perplexity-User PerplexityIts operator says this fetcher may not follow robots.txt.BlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
meta-externalfetcher MetaIts operator says this fetcher may not follow robots.txt.BlockedUser-agent: * (line 2)Disallow: /account/ (line 3)

Errors and warnings

  • WarningLine 14.Disalow is not a robots.txt directive, so crawlers ignore this line. Did you mean Disallow?
  • WarningLine 11.The OAI-SearchBot group does not repeat Disallow: /account/, Disallow: /checkout/ from the User-agent: * group. A crawler follows only the most specific group that names it, so OAI-SearchBot may fetch those paths. Copy the rules into its group if they should apply.
  • NoteLine 15.Crawl-delay is not part of RFC 9309. Google ignores it; Bing reads it. It never blocks a path.
  • NoteLine 17.Sitemap lines are an extension RFC 9309 allows, not a rule. They apply to the whole file wherever they appear and do not end a group.

Sitemaps declared

  • Line 17: https://basecamp-supply.example/sitemap.xml

Example: a robots.txt with three common mistakes

This file belongs to an outdoor-gear store that wants its account pages private and GPTBot blocked. Checked at /account/orders, OAI-SearchBot is allowed because its group replaces the * group, the misspelled Disalow line does nothing, and Crawl-delay blocks nothing.

robots.txt
# robots.txt for basecamp-supply.example
User-agent: *
Disallow: /account/
Disallow: /checkout/
Allow: /account/help

User-agent: GPTBot
User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Disalow: /cart
Crawl-delay: 5

Sitemap: https://basecamp-supply.example/sitemap.xml

18 of 19 crawlers blocked from /account/orders · 3 groups · 0 errors, 2 warnings

Results for /account/orders
CrawlerResultGroup usedDeciding rule
Search engines
Googlebot GoogleBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
bingbot MicrosoftBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
Applebot AppleBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
DuckDuckBot DuckDuckGoBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
AI training
GPTBot OpenAIBlockedUser-agent: GPTBot (line 7)Disallow: / (line 9)
ClaudeBot AnthropicBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
Google-Extended GoogleControl token: never crawls, so robots.txt is the only place to set it.BlockedUser-agent: Google-Extended (line 7)Disallow: / (line 9)
Applebot-Extended AppleControl token: never crawls, so robots.txt is the only place to set it.BlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
meta-externalagent MetaBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
Amazonbot AmazonBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
Bytespider ByteDanceBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
CCBot Common CrawlBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
AI search and indexing
OAI-SearchBot OpenAIAllowedUser-agent: OAI-SearchBot (line 11)Allow: / (line 12)
Claude-SearchBot AnthropicBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
PerplexityBot PerplexityBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
User-initiated fetches
ChatGPT-User OpenAIIts operator says this fetcher may not follow robots.txt.BlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
Claude-User AnthropicBlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
Perplexity-User PerplexityIts operator says this fetcher may not follow robots.txt.BlockedUser-agent: * (line 2)Disallow: /account/ (line 3)
meta-externalfetcher MetaIts operator says this fetcher may not follow robots.txt.BlockedUser-agent: * (line 2)Disallow: /account/ (line 3)

Errors and warnings

  • WarningLine 14.Disalow is not a robots.txt directive, so crawlers ignore this line. Did you mean Disallow?
  • WarningLine 11.The OAI-SearchBot group does not repeat Disallow: /account/, Disallow: /checkout/ from the User-agent: * group. A crawler follows only the most specific group that names it, so OAI-SearchBot may fetch those paths. Copy the rules into its group if they should apply.
  • NoteLine 15.Crawl-delay is not part of RFC 9309. Google ignores it; Bing reads it. It never blocks a path.
  • NoteLine 17.Sitemap lines are an extension RFC 9309 allows, not a rule. They apply to the whole file wherever they appear and do not end a group.

Sitemaps declared

  • Line 17: https://basecamp-supply.example/sitemap.xml

How a crawler reads robots.txt

This checker follows RFC 9309, the Robots Exclusion Protocol standard, and Google’s robots.txt specification, which documents how Google applies it. For every crawler and path, it works through the same steps a compliant crawler does:

  1. Pick the group. The crawler looks for groups whose User-agent line names its product token, ignoring case, and combines all of them. Only when none names it does it fall back to the User-agent: * group. With neither, every path is allowed.
  2. Find every matching rule. Each Allow and Disallow path is compared to the start of the URL path and query. * matches any run of characters and a $ at the end anchors the match to the end of the URL.
  3. Let the longest rule win. The matching rule with the most characters decides. When an Allow and a Disallow are the same length, Allow wins. No match means the path is allowed.
  4. Apply the file’s status. A 4xx response means there is no file, so everything is allowed. A server error means everything is disallowed until the file can be read; Google treats 429 the same way. /robots.txt itself is always allowed.
RuleMatchesDoes not match
Disallow: /account//account/, /account/orders/account, /Account/
Disallow: /*.pdf$/files/guide.pdf/files/guide.pdf?v=2
Disallow: /search*q=/search?q=tents/search
Disallow: (empty)No pathAny path: an empty Disallow allows everything

Paths are case-sensitive; user-agent tokens are not. Crawl-delay and Sitemap lines are reported but never block anything: Sitemap is an extension RFC 9309 allows, and Google ignores Crawl-delay.

Why AI crawlers get different answers

Each AI operator runs separate crawlers for separate jobs, and each has its own token. OpenAI’s GPTBot collects training data while OAI-SearchBot indexes pages for ChatGPT search, so one file can block one and allow the other. The results table groups every crawler by what it does.

The most common surprise is a named group replacing the * group. Once a file has User-agent: OAI-SearchBot, that crawler reads only its own group, including none of the Disallow lines you wrote for everyone else. The checker flags every named group that drops a path the * group blocks.

robots.txt: OAI-SearchBot may fetch /account/
User-agent: *
Disallow: /account/

User-agent: OAI-SearchBot
Allow: /

Two tokens never crawl. Google-Extended and Applebot-Extended set whether Google’s and Apple’s existing crawls may be used for AI, so robots.txt is the only place they take effect. Apple also documents that Applebot follows the Googlebot group when no group names Applebot, and the checker applies that rule.

Fetchers that may not follow robots.txt

Some assistants fetch a page because a person asked about it. OpenAI says ChatGPT-User may not follow robots.txt rules, Perplexity says Perplexity-User generally ignores them, and Meta says meta-externalfetcher may bypass them. The checker shows what your file asks of these fetchers and marks them, so you know where a block also needs a CDN or firewall rule.

Anthropic’s ClaudeBot, Claude-SearchBot and Claude-User all follow robots.txt. For user agents, IP ranges and verification, see the guides to ClaudeBot, OAI-SearchBot and PerplexityBot.

What the robots.txt validator flags

FindingWhat happens to the line
A rule before any User-agent lineIt belongs to no group, so every crawler ignores it.
An unknown or misspelled directive, such as DisalowIgnored. The checker suggests the directive you probably meant.
A path that does not start with /It matches no URL.
A named group that drops the * group’s pathsThat crawler may fetch paths you blocked for everyone else.
A group with no Allow or Disallow linesThe crawlers it names may fetch everything.
A byte order mark, invalid UTF-8 or a UTF-16 fileSome parsers misread the first line or every line.
A file over 500 KiBRFC 9309 only requires crawlers to read the first 500 KiB, and Google stops there.
An HTML page served at /robots.txtNo valid rules, so everything is allowed.
noindex in robots.txtIgnored. Use a robots meta tag or X-Robots-Tag header.

Fix what the checker finds

To rewrite the AI section of your file, the AI robots.txt generator builds one group per crawler by purpose and copies your private paths into each group, so the * group problem cannot happen. Check the result here before you publish it.

The crawlers you allow should find pages worth reading. The Rebilder gateway answers recognized AI agents at the same URL with clean markdown built from your own product, service and policy data, while people and search crawlers keep your HTML.

Frequently asked questions

How do I check my robots.txt file?

Choose “Fetch from a site”, enter your domain, and the checker reads https://your-domain/robots.txt. Or paste the file. Enter a path to see whether each search and AI crawler may fetch it, plus every error and warning in the file.

How do I test whether a URL is blocked by robots.txt?

Enter the URL or its path in “Path or URL to test”. For each crawler, the results show Allowed or Blocked, the group it used and the rule that decided.

Does Allow or Disallow win when both match?

The longest matching rule wins. When an Allow and a Disallow rule are the same length, Allow wins. That is RFC 9309’s rule and Google’s.

Why is a crawler allowed when my * group disallows the path?

Because a group names it. A crawler follows only the groups that name its token and ignores User-agent: * entirely, so repeat the * rules inside every named group.

What is the robots.txt size limit?

RFC 9309 requires crawlers to read at least 500 KiB, and Google reads exactly 500 KiB. Rules past that point may be ignored, so the checker flags a larger file.

Do AI crawlers follow robots.txt?

Operators document robots.txt as the way to opt out of their training and search crawlers. OpenAI, Perplexity and Meta say their user-initiated fetchers (ChatGPT-User, Perplexity-User, meta-externalfetcher) may not follow it, so the checker marks them.

Is my robots.txt file stored?

No. A pasted file is checked in your browser and is never sent to us. When you fetch from a site, we request that site’s /robots.txt once and show you the result without storing or publishing it.

Is this robots.txt checker free?

Yes: free, with no account.