---
title: "Cloudflare AI Crawl Control: allow, block and check every AI crawler"
description: "How Cloudflare AI Crawl Control works: per-crawler allow and block, plan limits, managed robots.txt, pay per crawl, WAF rules and what each bot receives."
canonical_url: https://rebilder.com/learn/cloudflare-ai-crawl-control
last_updated: "2026-11-15"
---

# [Cloudflare AI Crawl Control: allow, block and check every AI crawler](https://rebilder.com/learn/cloudflare-ai-crawl-control)

> How Cloudflare AI Crawl Control works: per-crawler allow and block, plan limits, managed robots.txt, pay per crawl, WAF rules and what each bot receives.

- **Updated:** 2026-11-15
- **Author:** Rebilder
- **Published:** 2026-11-15

**Cloudflare AI Crawl Control is the part of the Cloudflare dashboard that shows which AI crawlers request your site and lets you allow or block each one.** It was previously called AI Audit, it is available on every Cloudflare plan, and a block you set there is enforced at Cloudflare’s edge with a WAF custom rule rather than left to the crawler’s good manners.

This guide covers plans, per-crawler allow and block, how it relates to your `robots.txt` and Cloudflare’s managed `robots.txt`, pay per crawl, WAF custom rules, limits, and how to confirm what each AI crawler actually receives. The last section covers what the crawlers you allow get to read.

## What is Cloudflare AI Crawl Control?

Cloudflare describes AI Crawl Control as a way to “monitor and control how AI services access your website content.” It works on any domain whose traffic is proxied through Cloudflare, needs no setup to start reporting, and lives under **AI Crawl Control** in the zone dashboard. The product is organized into tabs:

- **Overview.** Request volume, top status code, top path and crawlers grouped by operator.
- **Crawlers.** One row per crawler with its operator, category, requests, `robots.txt` violations and an Allow or Block action.
- **Metrics.** Requests, status codes, data transfer, content types requested versus served, and top paths.
- **Directives.** Formerly Robots.txt: your file’s health, its Content Signals, and crawlers that ignore it.

Cloudflare’s bot reference for AI Crawl Control lists, among others, these crawlers and the category it assigns each. The split by purpose drives most decisions, as in the [AI agent user agents reference](/learn/ai-agent-user-agents):

| Operator | Crawler | Cloudflare category |
| --- | --- | --- |
| OpenAI | `GPTBot` / `OAI-SearchBot` / `ChatGPT-User` | AI Crawler / AI Search / AI Assistant |
| Anthropic | `ClaudeBot` / `Claude-SearchBot` / `Claude-User` | AI Crawler / AI Search / AI Assistant |
| Perplexity | `PerplexityBot` / `Perplexity-User` | AI Search / AI Assistant |
| Meta | `meta-externalagent` / `meta-externalfetcher` | AI Crawler / AI Assistant |
| Google | `Google-CloudVertexBot` | AI Crawler |
| Amazon | `Amazonbot` | AI Crawler |
| ByteDance | `Bytespider` | AI Crawler |
| Common Crawl | `CCBot` | AI Crawler |
| Apple | `Applebot` | AI Search |

Two tokens you will see in `robots.txt` guides are missing from that table on purpose: `Google-Extended` and `Applebot-Extended`. Google and Apple both document them as control tokens that never arrive as a user agent of their own, so there is no request for a firewall to match. You control them only in `robots.txt`.

## AI Crawl Control plans and availability

Every plan gets the dashboard and the allow or block switch. What changes with the plan is how crawlers are detected, how far back the analytics go, and the extras around a block:

| Capability | Free | Paid plans | Enterprise with Bot Management |
| --- | --- | --- | --- |
| Crawler detection | By user agent string: well-known, self-identifying crawlers | By user agent string | Bot Management detection IDs as well |
| Analytics window | Metrics tab covers the last day | The last day | Configurable timeframes |
| Allow and block per crawler | Yes | Yes | Yes |
| Custom block response (status and body) | No | Yes | Yes |
| AI referral metrics | No | Yes | Yes |

Charging crawlers through pay per crawl is listed in Cloudflare’s plan comparison alongside Bot Management on Enterprise, and the feature itself is in private beta. Two related features sit next to AI Crawl Control with their own plan rules. Managed `robots.txt` is available on all plans. Redirects for AI Training, which sends verified training crawlers from a duplicate page to its canonical URL, is available on Pro, Business and Enterprise.

## How to allow or block an AI crawler

1. Open the Cloudflare dashboard, select your account and domain, and go to **AI Crawl Control**.
2. Open the **Crawlers** tab and review which AI crawlers reach your domain and how many requests each one makes.
3. In the **Action** column, choose **Allow** or **Block** for each crawler.
4. On a paid plan, open **Settings** and edit **Block response** to choose the status code and write a plain-text body that blocked crawlers receive.

Cloudflare suggests allowing crawlers that give you value through citations, referrals or agreements. For most businesses that means treating the purposes separately: block training if you choose to, and keep the search indexers and user-initiated fetchers that put your pages in front of customers asking an assistant about you. The [ClaudeBot](/learn/claudebot) and [PerplexityBot](/learn/perplexitybot) guides walk through that choice for each operator.

Block response codes AI Crawl Control can return

```text
403 Forbidden            You do not want the crawler to access the content
402 Payment Required     The crawler must pay to access the content
```

Use the response body for a licensing contact or a link to your terms. Cloudflare also documents a Response Header Transform Rule that adds a `Link` header with `rel="license"` to blocked responses.

## AI Crawl Control vs robots.txt

The two do different jobs. `robots.txt` states your preference, and a crawler that honors it stays out. AI Crawl Control enforces a decision at the edge, whether or not the crawler reads your file. Cloudflare’s own guidance is to use `robots.txt` to express preferences and AI Crawl Control to enforce crawler-specific blocks.

- **Your file still matters.** Crawlers read it for your terms. OpenAI, Perplexity and Meta say their user-initiated fetchers may not follow it, so the edge block is the backstop.
- **The Directives tab watches the file.** It shows how often each hostname’s `robots.txt` is requested, its status and whether it has Content Signals. Errors on a file that exists usually mean a security rule is blocking crawlers from reading it.
- **It lists violations.** The violations table shows crawlers that requested paths your `robots.txt` disallows, with the exact directive that matched. Cloudflare compares past requests against your **current** file, so a new `Disallow` rule makes earlier, legitimate requests to that path appear as violations.

A named group in `robots.txt` replaces the `User-agent: *` group for that crawler, so any path rules you keep under `*` must be repeated inside each named group. The free [AI robots.txt generator](/tools/robots-txt-generator) builds the file per crawler, grouped by purpose, and repeats your disallowed paths for you.

## Managed robots.txt and Content Signals

Cloudflare can write AI directives into your `robots.txt` for you. Under **Security > Settings > AI Crawler and Agent access policies**, you set an action for three behaviors, Search, Agent and Training, and turn on **Enable Bot Preference Sync**. When at least one policy blocks or disallows, Cloudflare prepends a managed block to the `robots.txt` your origin serves, or creates the file if you have none. With every policy set to allow, the file is served unchanged.

robots.txt: the managed section Cloudflare prepends (Training set to Disallow)

```text
# BEGIN Cloudflare Managed content

User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

# END Cloudflare Managed Content

# ...your origin’s robots.txt follows
```

- **Content Signals** add a machine-readable usage line: `search` for a search index, `ai-input` for real-time AI answers, and `ai-train` for training or fine-tuning.
- **Disallow for Training also blocks at the edge** when Bot Preference Sync is on. Mixed-purpose crawlers used for both search and training are excluded from that edge block.
- **Check your zone’s current setting.** Cloudflare has changed the defaults for new domains over time, so confirm what your zone is set to rather than assuming.

The managed block sits above your own rules and contains its own `User-agent: *` group. If you need different rules per crawler, path exceptions, or search and user fetchers listed explicitly, write the AI groups in your origin file and leave the managed policies on allow.

## How Cloudflare pay per crawl works

Pay per crawl adds a third action next to Allow and Block: **Charge**. Cloudflare documents it as a private beta; you join through Cloudflare’s pay per crawl signup page or, on Enterprise, through your account team. As currently documented:

- **One price per zone** for each successful retrieval, applied to every crawler set to Charge. Dynamic pricing lets your origin or a Worker price individual content.
- **Crawlers pay or get a price.** A crawler sends payment intent in request headers and gets the page, or gets a Payment Required response with pricing. Cloudflare is the Merchant of Record.
- **Only successes are billed,** and re-crawling a page is charged again.
- **Some paths are always free,** including `/robots.txt` and `/sitemap.xml`, and you can exempt URI patterns with a Configuration Rule.
- **Blocks win.** WAF custom rules run first and Cloudflare bot solutions run before pay per crawl, so a WAF block or an AI bot block policy overrides Charge. Turn off any blanket AI bot block before charging.

## Finer control with WAF custom rules

When you block crawlers in AI Crawl Control, Cloudflare creates one WAF custom rule named **AI Crawl Control** and keeps it updated. Allowing every crawler uses no rule at all. You can extend that rule under **Security > Security rules**, filtered to custom rules, and Cloudflare preserves your additions when you change crawler actions later. Typical extensions:

- A path exception, so a blocked crawler can still read one section of the site.
- Extra user agents, or detection IDs on Bot Management, for crawlers the dashboard does not list.
- A hostname clause, so the block applies to one subdomain only.

WAF custom rule expressions (Rules language)

```text
# Block GPTBot everywhere except the public blog
(http.user_agent contains "GPTBot" and not starts_with(http.request.uri.path, "/blog/"))

# Block verified bots Cloudflare categorizes as AI training crawlers on one hostname
(cf.verified_bot_category eq "AI Crawler" and http.host eq "shop.example.com")
```

Order matters. Custom rules run in sequence and a block stops evaluation, and the AI Crawl Control rule is added after your existing rules. If a crawler you set to Allow is still blocked, look for an earlier custom rule that blocks it. If a crawler you set to Block still gets through, look for skip, redirect or transform rules that run first, and move the AI Crawl Control rule to the top. Edits made directly in the WAF are not reflected back in the AI Crawl Control dashboard.

To tell a real crawler from a scraper that copied its name, use Cloudflare’s **verified bot** signals. Cloudflare verifies a bot that identifies itself deterministically, through a Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS, and that obeys `robots.txt` and keeps reasonable request rates. In rules, `cf.client.bot` is true for known good bots and `cf.verified_bot_category` carries categories such as `AI Crawler`, `AI Assistant` and `AI Search`.

## Limits to plan around

- **Rule budget.** Each plan has a maximum number of WAF custom rules; the Free plan allows five. AI Crawl Control uses one of them as soon as you block a crawler, and regular expressions in custom rules need a Business or Enterprise plan.
- **User-agent detection is a claim.** Without Bot Management, a crawler using a browser user agent is missed, and anything copying a listed token is treated as that crawler.
- **One-day history without Bot Management.** Export from the Metrics tab or use the GraphQL Analytics API.
- **Upstream rules hide traffic.** Requests stopped before the AI Crawl Control rule may not show in its analytics.
- **robots.txt-only tokens.** `Google-Extended` and `Applebot-Extended` cannot be blocked at the edge.

## How to check what each AI crawler actually receives

Setting a crawler to Allow tells you it gets through. It does not tell you what it reads. Check both layers:

1. In **Metrics**, filter to one crawler and read the status code distribution. A crawler you allowed should be getting successes, not a wall of challenges, redirects or errors.
2. In the same tab, open **Content Format** to compare the content types crawlers request, by `Accept` header, with what your origin serves, by `Content-Type`.
3. Use **Most popular paths** with the crawler filter to see which pages it fetches, and **Security Analytics** to see which rule acted.
4. Fetch a key page with the crawler’s token and with a markdown `Accept` header. Is the price, service detail or policy in the response body, or only in content a script assembles later?

See the response a crawler gets

```bash
# Status and headers for a page, as GPTBot
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  https://your-site.example/products/example

# The body an agent gets when it asks for markdown
curl -s -H "Accept: text/markdown" https://your-site.example/products/example | head -40

# Your live robots.txt, from outside your network
curl -s https://your-site.example/robots.txt
```

On plans that detect by user agent, a request carrying a blocked token gets the same block response the crawler gets, which makes the first command a quick test of your rule. With Bot Management, detection uses more than the header, so a request from your own machine may be treated differently from the real crawler; rely on the Metrics tab there.

## Serve the crawlers you allow clean markdown with Rebilder

AI Crawl Control decides who gets in. Rebilder handles what the agents you let in read. The gateway sits in front of your site, recognizes agent requests and answers them with markdown rendered from your own product, service and policy data, while browsers and search crawlers keep getting your HTML at the same URL.

- **It runs on Cloudflare.** The [Cloudflare Worker adapter](/docs/adapters/cloudflare) deploys the gateway on a route in front of a site already on Cloudflare, on any plan including Free. Anything it does not serve passes through to your origin unchanged.
- **It recognizes the agents AI Crawl Control lists.** The open-source detector matches `GPTBot`, `OAI-SearchBot`, `ChatGPT-User`, `ClaudeBot`, `Claude-SearchBot`, `Claude-User`, `PerplexityBot`, `Perplexity-User`, `meta-externalagent`, `meta-externalfetcher`, `Amazonbot`, `DuckAssistBot`, `MistralAI-User` and `Bytespider`, and any request that sends `Accept: text/markdown`. `Applebot`, `Googlebot` and `bingbot` are search crawlers and always get your canonical HTML. See [Classification](/docs/classification).
- **It renders from your sources, not your HTML.** A recognized agent gets markdown when one of your configured sources answers the URL, with the facts at the top; a URL without a source falls through to your normal page. Cloudflare’s Markdown for Agents converts your HTML when a client asks for markdown; the gateway serves the data you connected, including to recognized agents that do not ask.
- **It reports what each agent received.** Connect an event sink and the Console shows agent visits by platform and whether each one got sourced markdown or your regular page. Pair that with AI Crawl Control’s view of who was blocked.

The gateway also has an [access policy](/docs/access-control) to allow, deny or rate-limit agents. On Cloudflare, keeping crawler blocks in AI Crawl Control and letting the gateway serve is the simpler split.

## What is Cloudflare AI Crawl Control?

A Cloudflare dashboard feature, formerly AI Audit, that shows which AI crawlers request your site and lets you allow or block each one. Blocks are enforced at Cloudflare’s edge with a WAF custom rule. It is available on all plans.

## Is AI Crawl Control free?

Yes. Every plan, including Free, gets the dashboard, user-agent based detection and per-crawler allow and block. Paid plans add a custom block response and referral metrics; Enterprise with Bot Management adds detection IDs and configurable analytics timeframes.

## Does AI Crawl Control replace robots.txt?

No. `robots.txt` states your preference and AI Crawl Control enforces a block at the edge. Keep both: crawlers read your file for your terms, the Directives tab reports crawlers that ignore it, and `Google-Extended` and `Applebot-Extended` can only be controlled in `robots.txt`.

## How do I block GPTBot on Cloudflare?

Go to **AI Crawl Control > Crawlers**, find `GPTBot` and set its action to **Block**. Add `User-agent: GPTBot` with `Disallow: /` to your `robots.txt` as well. Blocking GPTBot does not block `OAI-SearchBot` or `ChatGPT-User`; set those separately.

## What is Cloudflare pay per crawl?

An AI Crawl Control action, in private beta, that charges a crawler for each successful page retrieval at a single price you set for the zone. Crawlers send payment intent in headers or receive a Payment Required response with pricing, and Cloudflare acts as the Merchant of Record.

## Why is a crawler I allowed still blocked?

An earlier WAF custom rule, a bot setting that blocks AI bots, or an AI Crawler and Agent access policy is likely stopping it before the AI Crawl Control rule runs. Check **Security > Security rules** and **Security > Settings**, and look the request up in Security Analytics to see which rule acted.

## Does Cloudflare block AI crawlers by default?

It depends on your zone’s AI Crawler and Agent access policies and managed `robots.txt` settings, and Cloudflare has changed the defaults for new domains over time. Open **Security > Settings > AI Crawler and Agent access policies** to see what your zone does today.

## Related

- [Free AI robots.txt generator](https://rebilder.com/tools/robots-txt-generator)
- [AI agent user agents: the full reference](https://rebilder.com/learn/ai-agent-user-agents)
- [ClaudeBot: user agent, robots.txt and verification](https://rebilder.com/learn/claudebot)
- [PerplexityBot: user agent, robots.txt and IPs](https://rebilder.com/learn/perplexitybot)
- [Deploy the gateway as a Cloudflare Worker](https://rebilder.com/docs/adapters/cloudflare)
- [How agent detection works](https://rebilder.com/docs/classification)