---
title: "ClaudeBot: what Anthropic’s crawler does and how to control it"
description: ClaudeBot is Anthropic’s training crawler. Its user agent, robots.txt rules for ClaudeBot, Claude-User and Claude-SearchBot, and how to verify the traffic.
canonical_url: https://rebilder.com/learn/claudebot
last_updated: "2026-11-15"
---

# [ClaudeBot: what Anthropic’s crawler does and how to control it](https://rebilder.com/learn/claudebot)

> ClaudeBot is Anthropic’s training crawler. Its user agent, robots.txt rules for ClaudeBot, Claude-User and Claude-SearchBot, and how to verify the traffic.

- **Updated:** 2026-11-15
- **Author:** Rebilder
- **Published:** 2026-11-15

**ClaudeBot is Anthropic’s web crawler: it collects public web pages that may be used to train and improve Anthropic’s Claude models.** It identifies itself with the `ClaudeBot` user-agent token, honors `robots.txt`, and is one of three Anthropic bots you control separately, alongside `Claude-User` (fetches a page when a person asks Claude about it) and `Claude-SearchBot` (indexes pages for Claude’s search results).

If you found ClaudeBot in your access logs, this guide covers what each Anthropic bot does with your content, the exact `robots.txt` rules to allow or block each one, how to confirm a request really came from Anthropic, and what you give up when you block it. For the wider family of AI user agents, see the [AI agent user agents reference](/learn/ai-agent-user-agents).

## What is ClaudeBot?

> **Looking for Clawdbot?** The open-source personal assistant once called Clawdbot is a separate project, now named OpenClaw. This guide covers ClaudeBot, Anthropic’s web crawler.

ClaudeBot is the crawler Anthropic runs to gather publicly available web content for model training. Anthropic’s own help article, “Does Anthropic crawl data from the web, and how can site owners block the crawler?”, describes it as helping “enhance the utility and safety of our generative AI models,” and says that blocking it signals that your site’s future materials should be excluded from Anthropic’s training datasets.

Anthropic’s own documentation, for the live details

```text
Crawlers and robots.txt:  https://support.claude.com/en/articles/8896518-does-anthropic-crawl-the-web
Hosted browsers:          https://support.claude.com/en/articles/18234133
Bot IP addresses:         https://claude.com/crawling/bots.json
```

The crawler article sets out the principles Anthropic applies to all of its bots:

- **It honors `robots.txt`.** Anthropic’s bots respect “do not crawl” signals expressed through industry-standard `robots.txt` directives.
- **It supports `Crawl-delay`.** Anthropic explicitly supports this non-standard extension, so you can slow ClaudeBot down without blocking it.
- **It does not bypass CAPTCHAs.** Pages behind a challenge stay behind it.
- **It is one of three bots.** ClaudeBot handles training collection only. Live answers and Claude’s search index use different user agents with their own `robots.txt` tokens.

That last point drives most decisions. ClaudeBot is the bot to block if your concern is training. It is not the bot that decides whether Claude can read your page when a customer asks about it.

## ClaudeBot vs Claude-User vs Claude-SearchBot

Anthropic names three bots in its crawler documentation and gives each its own consequence when blocked. A separate article covers browsers Anthropic hosts for Claude to operate on a user’s behalf, which add a fourth token:

| Bot | Role | Honors robots.txt | robots.txt token | If you block it |
| --- | --- | --- | --- | --- |
| **ClaudeBot** | Collects public web content that may be used to train Anthropic’s models. | Yes | `ClaudeBot` | Your future content is excluded from Anthropic’s training datasets. |
| **Claude-User** | Fetches a page when a Claude user asks a question that needs it. | Yes | `Claude-User` | Claude cannot retrieve your page in response to a user’s question, which may reduce your visibility for user-directed web search. |
| **Claude-SearchBot** | Indexes content to improve the relevance and accuracy of Claude’s search results. | Yes | `Claude-SearchBot` | Your content is not indexed for Claude’s search, which may reduce your visibility and accuracy in its results. |
| **Claude-Agent** | A real browser on Anthropic’s infrastructure, driven by a hosted Claude agent completing a task for a user. | Not covered by the crawler article; Anthropic manages it separately. | None documented. Match the `Claude-Agent` user-agent token plus Anthropic’s IP list at your firewall. | Users cannot ask Claude’s hosted agent to complete tasks on your site. |

The pattern mirrors OpenAI’s split between `GPTBot`, `OAI-SearchBot` and `ChatGPT-User`, covered in [GPTBot: allow it, block it, or serve it properly](/solutions/gptbot). One bot builds a training corpus, one builds a search index, and one acts on a live request from a person.

> **Older tokens** Older `robots.txt` guides list `anthropic-ai` and `Claude-Web`. Anthropic’s current crawler article names only `ClaudeBot`, `Claude-User` and `Claude-SearchBot`. Keeping a rule for the older tokens costs nothing, but it is the three current tokens that Anthropic documents.

## ClaudeBot user agent string

Anthropic documents the user-agent **token** (`ClaudeBot`), which is what `robots.txt` matches on. Its crawler article does not print a full user-agent string or version. The full string as it commonly appears in access logs looks like this:

ClaudeBot user agent as commonly logged

```http
User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
```

The contact address in that string is the one Anthropic publishes for crawler questions and reports. Anthropic asks that you write from an address on the domain you are asking about, so it can match the report to the site.

For hosted browsers, Anthropic’s article “How do Anthropic-hosted browsers identify themselves when they browse the web on a user’s behalf?” shows a normal Chrome user agent with a `Claude-Agent` token added, and tells site owners to match the token rather than the full string, because other parts of it can change:

Claude-Agent example from Anthropic’s article

```http
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-Agent/1.0) Chrome/154.0.0.0 Safari/537.36
```

In every case, match on the token, case-insensitively, and never pin a version number. Version suffixes change, and a regex built for one exact string silently stops matching.

## How to block ClaudeBot in robots.txt

Anthropic’s documented opt-out is a `robots.txt` group naming ClaudeBot. Put it in the `robots.txt` at the root of **every subdomain** you want to opt out, because each host serves its own file:

robots.txt: block ClaudeBot (training) only

```text
User-agent: ClaudeBot
Disallow: /
```

That rule blocks training collection and nothing else. `Claude-User` and `Claude-SearchBot` read their own groups, so they keep fetching unless you name them too. To block every Anthropic bot:

robots.txt: block all Anthropic bots

```text
User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: Claude-SearchBot
Disallow: /
```

Most businesses whose product is not the text itself, such as a store, a clinic or a services firm, are better served by declining training while staying answerable when a customer asks Claude about them. That is a supported configuration:

robots.txt: block training, allow search and user fetches

```text
User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /
```

You can also scope a block to paths rather than the whole site, for example to keep crawlers out of account and checkout flows while leaving product and policy pages readable:

robots.txt: block private paths for every Anthropic bot

```text
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /account/
Disallow: /checkout/
Disallow: /cart
```

> **A named group replaces your wildcard rules** Under the robots exclusion standard, a crawler follows the group that names its token and ignores the `User-agent: *` group. If your `*` group disallows `/admin/` and you add a `User-agent: Claude-SearchBot` group with only `Allow: /`, Claude-SearchBot is no longer told to stay out of `/admin/`. Repeat any path rules you still want inside each named group.

Check the live file after you deploy: fetch `/robots.txt` from outside your network, confirm it returns plain text without a login or challenge page, and confirm your CDN is not serving a cached older copy.

## ClaudeBot crawl rate and Crawl-delay

If ClaudeBot is fetching more than your server comfortably handles, slow it down instead of blocking it. Anthropic supports the `Crawl-delay` extension and gives this example in its help article:

robots.txt: slow ClaudeBot down

```text
User-agent: ClaudeBot
Crawl-delay: 1
```

Raise the value to spread requests further apart. `Crawl-delay` is not part of the core robots standard, and other crawlers may ignore it, so set it per bot rather than relying on a `*` group. For hard limits, add a rate-limit rule at your CDN or server keyed on the user-agent token, which caps requests whatever any crawler chooses to honor.

## How to verify a request is really ClaudeBot

Any client can send `ClaudeBot` in its user agent, so the string alone proves nothing. Anthropic publishes the source IP addresses its bots use:

Fetch Anthropic’s published bot IP ranges

```bash
curl -s https://claude.com/crawling/bots.json
# {
#   "creationTime": "...",
#   "prefixes": [ { "ipv4Prefix": "..." }, ... ]
# }
```

- **If the source IP is in that list, the request came from Anthropic.** That is how Anthropic’s article phrases it. A request that claims to be ClaudeBot from an address outside the list is someone else using the name.
- **Read the list on a schedule.** Prefixes change. Load the file into your firewall’s IP list automatically rather than copying addresses by hand.
- **Hosted browsers use a separate list.** `Claude-Agent` traffic is published at `https://claude.com/.well-known/sail`, listed under that token.
- **Reverse DNS is not the documented method.** Anthropic documents the IP list, not a hostname pattern, so verify against the list.

Use the IP list to **identify** traffic, not to opt out of it. Anthropic advises against blocking its IP addresses as an opt-out: it may not work correctly or persistently, because it also stops ClaudeBot from reading the `robots.txt` that carries your opt-out. Block in `robots.txt`; use the IP list to separate genuine Anthropic traffic from impostors and treat impostors as you would any unidentified scraper.

Anthropic’s crawler documentation does not describe Web Bot Auth request signatures for ClaudeBot, Claude-User or Claude-SearchBot. Today, the IP list is the verification method it documents.

## What happens if you block ClaudeBot

Blocking ClaudeBot stops Anthropic from collecting your future pages for training. It does not remove you from Claude’s answers, because answers and search use the other two bots. Here is the practical effect of each choice:

- **Block ClaudeBot only.** Anthropic excludes your future materials from its training datasets. Claude can still find you through search and read your page when a user asks.
- **Block Claude-SearchBot.** Claude’s search index no longer includes your pages, so Claude is less likely to surface or cite you when users search.
- **Block Claude-User.** When a user pastes your URL or asks something only your page answers, Claude cannot fetch it. The user gets an answer without your page in it.
- **Block all three.** Your site is effectively invisible to Claude’s live answers and search, and excluded from future training.

Anthropic also now recognizes a machine-readable “no AI training” preference based on the IETF aipref working group’s draft-kelley-aipref-training-preference-signal. If your `robots.txt` names ClaudeBot, Anthropic follows that rule even where your general preference says something different; if it doesn’t, the general preference applies. A ClaudeBot group in `robots.txt` remains the most direct control.

## Does ClaudeBot render JavaScript?

Anthropic does not document whether ClaudeBot, Claude-User or Claude-SearchBot execute JavaScript. Plan for the safer case: put the facts that matter, such as prices, availability, hours, shipping and return terms, in the HTML the server returns, not only in content a script assembles after load.

You can check what a non-rendering client sees with a plain request. If the price or the answer is missing from the response body, a fetcher that doesn’t run scripts is missing it too:

See the raw HTML a crawler receives

```bash
curl -s -A "Mozilla/5.0 (compatible; ClaudeBot/1.0)" https://your-site.example/products/example | grep -i "price"
```

## How to find ClaudeBot in your logs and CDN

Start with your own access logs. These commands count requests per Anthropic bot and list the pages each one fetches most:

Count Anthropic bots in an access log

```bash
# Requests per Anthropic bot token
grep -ioE "claudebot|claude-user|claude-searchbot|claude-agent" /var/log/nginx/access.log \
  | tr "[:upper:]" "[:lower:]" | sort | uniq -c | sort -rn

# Most-fetched paths for ClaudeBot (combined log format: path is field 7)
grep -i "claudebot" /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

# Source IPs claiming to be ClaudeBot, to check against bots.json
grep -i "claudebot" /var/log/nginx/access.log | awk '{print $1}' | sort -u
```

- **Cloudflare.** AI Crawl Control lists the AI crawlers reaching your zone and lets you allow or block each one; a block is enforced with a WAF rule. For finer control, write a custom rule that matches the user-agent token together with Anthropic’s IP list.
- **Vercel.** The Firewall’s AI Bots managed ruleset can log or deny AI crawlers and assistants. Set it to log first, review what arrives, then decide.
- **Any other host.** Your CDN or load balancer logs carry the user agent and source IP. The same grep works on an exported log.

A CDN rule that denies a bot overrides your `robots.txt` choice for that bot. If you allow Claude-User in `robots.txt` but a firewall rule challenges every AI user agent, Claude still can’t read the page. Check both layers together.

## Serve Anthropic’s bots a clean answer with Rebilder

Allowing a bot is half the decision. The other half is what it reads. Rebilder’s gateway sits in front of your site, recognizes agent requests and answers them with markdown rendered from your own product, service and policy data, while browsers and search crawlers keep getting your HTML at the same URL.

- **Detection.** The open-source detector, `@rebilder/agent-detect`, matches the `ClaudeBot`, `Claude-User`, `Claude-SearchBot` and `Claude-Agent` tokens, plus the older `claude-web` and `anthropic-ai`, to the `claude` platform at medium confidence, because a user agent alone is a claim. The full order of checks is in [Classification](/docs/classification).
- **Serving.** A recognized AI agent gets markdown when one of your configured sources answers the URL; any URL without a source falls through to your normal HTML. Search crawlers such as Googlebot always get canonical HTML.
- **Access rules.** The gateway’s [access policy](/docs/access-control) can allow, deny or rate-limit agents. A rule naming the `claude` platform applies only to verified requests, so for user-agent-only traffic like ClaudeBot use the `unverified` or `*` subjects, and keep your `robots.txt` as the opt-out Anthropic reads.
- **Reporting.** Connect an event sink and the Console shows each request by platform, with what it received: markdown from a source, or your regular page. See [Which agents Rebilder detects](/help/agent-traffic/which-agents-detected).

The detector reads request headers; it does not check source IPs. Pair it with an IP-list rule at your CDN when you need to confirm a request came from Anthropic.

## What is ClaudeBot?

ClaudeBot is Anthropic’s web crawler. It collects public web content that may be used to train Anthropic’s Claude models, identifies itself with the `ClaudeBot` user-agent token, and honors `robots.txt`. It is separate from `Claude-User`, which fetches pages for live questions, and `Claude-SearchBot`, which indexes pages for Claude’s search.

## Is ClaudeBot safe?

Genuine ClaudeBot is a documented crawler run by Anthropic: it honors `robots.txt`, supports `Crawl-delay` and does not bypass CAPTCHAs. The risk is impostors, since anyone can copy the user agent. Check the source IP against Anthropic’s published list at `https://claude.com/crawling/bots.json`; a request outside that list is not Anthropic.

## How do I block ClaudeBot?

Add `User-agent: ClaudeBot` followed by `Disallow: /` to the `robots.txt` at the root of each subdomain. That blocks training collection only. To keep Claude from reading your pages entirely, add the same rule for `Claude-User` and `Claude-SearchBot`.

## Does blocking ClaudeBot remove my site from Claude’s answers?

No. ClaudeBot is the training crawler. Claude’s search results use `Claude-SearchBot` and live page fetches use `Claude-User`, each with its own `robots.txt` token. Block those two as well if you want Claude unable to read your site.

## Does ClaudeBot respect Crawl-delay?

Yes. Anthropic states that it supports the non-standard `Crawl-delay` extension. Add `Crawl-delay` under a `User-agent: ClaudeBot` group to slow it down instead of blocking it.

## What IP addresses does ClaudeBot use?

Anthropic publishes them at `https://claude.com/crawling/bots.json`. The list changes, so load it into your firewall on a schedule. Anthropic recommends using `robots.txt` to opt out, because blocking its IPs also stops the bot reading your opt-out.

## What are anthropic-ai and Claude-Web?

Older user-agent tokens that still appear in many `robots.txt` guides. Anthropic’s current crawler documentation names `ClaudeBot`, `Claude-User` and `Claude-SearchBot`, so write your rules for those three. Leaving rules for the older tokens in place does no harm.

## What is Claude-Agent?

The token Anthropic adds to the user agent of browsers it hosts for Claude to complete tasks for a user. It is managed separately from the crawlers: match the `Claude-Agent` token and Anthropic’s list at `https://claude.com/.well-known/sail` at your firewall to allow or block it.

## Related

- [AI agent user agents: the full reference](https://rebilder.com/learn/ai-agent-user-agents)
- [PerplexityBot: user agent, robots.txt and IPs](https://rebilder.com/learn/perplexitybot)
- [How agent detection works](https://rebilder.com/docs/classification)
- [Allow, deny or rate-limit agents](https://rebilder.com/docs/access-control)
- [Which agents Rebilder detects](https://rebilder.com/help/agent-traffic/which-agents-detected)
- [Gateway quickstart](https://rebilder.com/docs/quickstart)
- [Free AI robots.txt generator](https://rebilder.com/tools/robots-txt-generator)