Free tool
robots.txt checker and validator
Paste your robots.txt or fetch it from your site, then test any URL. See whether Googlebot, Bingbot and every major AI crawler may fetch it, which group and rule decided, and which lines crawlers will misread or ignore.
18 of 19 crawlers blocked from /account/orders · 3 groups · 0 errors, 2 warnings
| Crawler | Result | Group used | Deciding rule |
|---|---|---|---|
| Search engines | |||
| Googlebot Google | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| bingbot Microsoft | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| Applebot Apple | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| DuckDuckBot DuckDuckGo | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| AI training | |||
| GPTBot OpenAI | Blocked | User-agent: GPTBot (line 7) | Disallow: / (line 9) |
| ClaudeBot Anthropic | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| Google-Extended GoogleControl token: never crawls, so robots.txt is the only place to set it. | Blocked | User-agent: Google-Extended (line 7) | Disallow: / (line 9) |
| Applebot-Extended AppleControl token: never crawls, so robots.txt is the only place to set it. | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| meta-externalagent Meta | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| Amazonbot Amazon | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| Bytespider ByteDance | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| CCBot Common Crawl | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| AI search and indexing | |||
| OAI-SearchBot OpenAI | Allowed | User-agent: OAI-SearchBot (line 11) | Allow: / (line 12) |
| Claude-SearchBot Anthropic | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| PerplexityBot Perplexity | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| User-initiated fetches | |||
| ChatGPT-User OpenAIIts operator says this fetcher may not follow robots.txt. | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| Claude-User Anthropic | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| Perplexity-User PerplexityIts operator says this fetcher may not follow robots.txt. | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| meta-externalfetcher MetaIts operator says this fetcher may not follow robots.txt. | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
Errors and warnings
- WarningLine 14.
Disalowis not a robots.txt directive, so crawlers ignore this line. Did you meanDisallow? - WarningLine 11.The
OAI-SearchBotgroup does not repeatDisallow: /account/,Disallow: /checkout/from theUser-agent: *group. A crawler follows only the most specific group that names it, soOAI-SearchBotmay fetch those paths. Copy the rules into its group if they should apply. - NoteLine 15.Crawl-delay is not part of RFC 9309. Google ignores it; Bing reads it. It never blocks a path.
- NoteLine 17.Sitemap lines are an extension RFC 9309 allows, not a rule. They apply to the whole file wherever they appear and do not end a group.
Sitemaps declared
- Line 17: https://basecamp-supply.example/sitemap.xml
Example: a robots.txt with three common mistakes
This file belongs to an outdoor-gear store that wants its account pages private and GPTBot blocked. Checked at /account/orders, OAI-SearchBot is allowed because its group replaces the * group, the misspelled Disalow line does nothing, and Crawl-delay blocks nothing.
# robots.txt for basecamp-supply.example
User-agent: *
Disallow: /account/
Disallow: /checkout/
Allow: /account/help
User-agent: GPTBot
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
Disalow: /cart
Crawl-delay: 5
Sitemap: https://basecamp-supply.example/sitemap.xml
18 of 19 crawlers blocked from /account/orders · 3 groups · 0 errors, 2 warnings
| Crawler | Result | Group used | Deciding rule |
|---|---|---|---|
| Search engines | |||
| Googlebot Google | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| bingbot Microsoft | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| Applebot Apple | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| DuckDuckBot DuckDuckGo | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| AI training | |||
| GPTBot OpenAI | Blocked | User-agent: GPTBot (line 7) | Disallow: / (line 9) |
| ClaudeBot Anthropic | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| Google-Extended GoogleControl token: never crawls, so robots.txt is the only place to set it. | Blocked | User-agent: Google-Extended (line 7) | Disallow: / (line 9) |
| Applebot-Extended AppleControl token: never crawls, so robots.txt is the only place to set it. | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| meta-externalagent Meta | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| Amazonbot Amazon | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| Bytespider ByteDance | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| CCBot Common Crawl | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| AI search and indexing | |||
| OAI-SearchBot OpenAI | Allowed | User-agent: OAI-SearchBot (line 11) | Allow: / (line 12) |
| Claude-SearchBot Anthropic | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| PerplexityBot Perplexity | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| User-initiated fetches | |||
| ChatGPT-User OpenAIIts operator says this fetcher may not follow robots.txt. | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| Claude-User Anthropic | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| Perplexity-User PerplexityIts operator says this fetcher may not follow robots.txt. | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
| meta-externalfetcher MetaIts operator says this fetcher may not follow robots.txt. | Blocked | User-agent: * (line 2) | Disallow: /account/ (line 3) |
Errors and warnings
- WarningLine 14.
Disalowis not a robots.txt directive, so crawlers ignore this line. Did you meanDisallow? - WarningLine 11.The
OAI-SearchBotgroup does not repeatDisallow: /account/,Disallow: /checkout/from theUser-agent: *group. A crawler follows only the most specific group that names it, soOAI-SearchBotmay fetch those paths. Copy the rules into its group if they should apply. - NoteLine 15.Crawl-delay is not part of RFC 9309. Google ignores it; Bing reads it. It never blocks a path.
- NoteLine 17.Sitemap lines are an extension RFC 9309 allows, not a rule. They apply to the whole file wherever they appear and do not end a group.
Sitemaps declared
- Line 17: https://basecamp-supply.example/sitemap.xml
How a crawler reads robots.txt
This checker follows RFC 9309, the Robots Exclusion Protocol standard, and Google’s robots.txt specification, which documents how Google applies it. For every crawler and path, it works through the same steps a compliant crawler does:
- Pick the group. The crawler looks for groups whose
User-agentline names its product token, ignoring case, and combines all of them. Only when none names it does it fall back to theUser-agent: *group. With neither, every path is allowed. - Find every matching rule. Each
AllowandDisallowpath is compared to the start of the URL path and query.*matches any run of characters and a$at the end anchors the match to the end of the URL. - Let the longest rule win. The matching rule with the most characters decides. When an
Allowand aDisalloware the same length,Allowwins. No match means the path is allowed. - Apply the file’s status. A 4xx response means there is no file, so everything is allowed. A server error means everything is disallowed until the file can be read; Google treats 429 the same way.
/robots.txtitself is always allowed.
| Rule | Matches | Does not match |
|---|---|---|
Disallow: /account/ | /account/, /account/orders | /account, /Account/ |
Disallow: /*.pdf$ | /files/guide.pdf | /files/guide.pdf?v=2 |
Disallow: /search*q= | /search?q=tents | /search |
Disallow: (empty) | No path | Any path: an empty Disallow allows everything |
Paths are case-sensitive; user-agent tokens are not. Crawl-delay and Sitemap lines are reported but never block anything: Sitemap is an extension RFC 9309 allows, and Google ignores Crawl-delay.
Why AI crawlers get different answers
Each AI operator runs separate crawlers for separate jobs, and each has its own token. OpenAI’s GPTBot collects training data while OAI-SearchBot indexes pages for ChatGPT search, so one file can block one and allow the other. The results table groups every crawler by what it does.
The most common surprise is a named group replacing the * group. Once a file has User-agent: OAI-SearchBot, that crawler reads only its own group, including none of the Disallow lines you wrote for everyone else. The checker flags every named group that drops a path the * group blocks.
User-agent: *
Disallow: /account/
User-agent: OAI-SearchBot
Allow: /Two tokens never crawl. Google-Extended and Applebot-Extended set whether Google’s and Apple’s existing crawls may be used for AI, so robots.txt is the only place they take effect. Apple also documents that Applebot follows the Googlebot group when no group names Applebot, and the checker applies that rule.
Fetchers that may not follow robots.txt
Some assistants fetch a page because a person asked about it. OpenAI says ChatGPT-User may not follow robots.txt rules, Perplexity says Perplexity-User generally ignores them, and Meta says meta-externalfetcher may bypass them. The checker shows what your file asks of these fetchers and marks them, so you know where a block also needs a CDN or firewall rule.
Anthropic’s ClaudeBot, Claude-SearchBot and Claude-User all follow robots.txt. For user agents, IP ranges and verification, see the guides to ClaudeBot, OAI-SearchBot and PerplexityBot.
What the robots.txt validator flags
| Finding | What happens to the line |
|---|---|
A rule before any User-agent line | It belongs to no group, so every crawler ignores it. |
An unknown or misspelled directive, such as Disalow | Ignored. The checker suggests the directive you probably meant. |
A path that does not start with / | It matches no URL. |
A named group that drops the * group’s paths | That crawler may fetch paths you blocked for everyone else. |
A group with no Allow or Disallow lines | The crawlers it names may fetch everything. |
| A byte order mark, invalid UTF-8 or a UTF-16 file | Some parsers misread the first line or every line. |
| A file over 500 KiB | RFC 9309 only requires crawlers to read the first 500 KiB, and Google stops there. |
An HTML page served at /robots.txt | No valid rules, so everything is allowed. |
noindex in robots.txt | Ignored. Use a robots meta tag or X-Robots-Tag header. |
Fix what the checker finds
To rewrite the AI section of your file, the AI robots.txt generator builds one group per crawler by purpose and copies your private paths into each group, so the * group problem cannot happen. Check the result here before you publish it.
The crawlers you allow should find pages worth reading. The Rebilder gateway answers recognized AI agents at the same URL with clean markdown built from your own product, service and policy data, while people and search crawlers keep your HTML.
Frequently asked questions
How do I check my robots.txt file?
Choose “Fetch from a site”, enter your domain, and the checker reads https://your-domain/robots.txt. Or paste the file. Enter a path to see whether each search and AI crawler may fetch it, plus every error and warning in the file.
How do I test whether a URL is blocked by robots.txt?
Enter the URL or its path in “Path or URL to test”. For each crawler, the results show Allowed or Blocked, the group it used and the rule that decided.
Does Allow or Disallow win when both match?
The longest matching rule wins. When an Allow and a Disallow rule are the same length, Allow wins. That is RFC 9309’s rule and Google’s.
Why is a crawler allowed when my * group disallows the path?
Because a group names it. A crawler follows only the groups that name its token and ignores User-agent: * entirely, so repeat the * rules inside every named group.
What is the robots.txt size limit?
RFC 9309 requires crawlers to read at least 500 KiB, and Google reads exactly 500 KiB. Rules past that point may be ignored, so the checker flags a larger file.
Do AI crawlers follow robots.txt?
Operators document robots.txt as the way to opt out of their training and search crawlers. OpenAI, Perplexity and Meta say their user-initiated fetchers (ChatGPT-User, Perplexity-User, meta-externalfetcher) may not follow it, so the checker marks them.
Is my robots.txt file stored?
No. A pasted file is checked in your browser and is never sent to us. When you fetch from a site, we request that site’s /robots.txt once and show you the result without storing or publishing it.
Is this robots.txt checker free?
Yes: free, with no account.