---
title: "Bytespider: what ByteDance’s crawler is and how to block it"
description: Bytespider is ByteDance’s web crawler. What is documented about it, its user agent, and how to block it in robots.txt and at your CDN or web server.
canonical_url: https://rebilder.com/learn/bytespider
last_updated: "2026-11-15"
---

# [Bytespider: what ByteDance’s crawler is and how to block it](https://rebilder.com/learn/bytespider)

> Bytespider is ByteDance’s web crawler. What is documented about it, its user agent, and how to block it in robots.txt and at your CDN or web server.

- **Updated:** 2026-11-15
- **Author:** Rebilder
- **Published:** 2026-11-15

**Bytespider is the web crawler run by ByteDance, the company that owns TikTok and the Toutiao news app.** It announces itself with a `Bytespider` token in the user-agent header, and that token is what you name in `robots.txt` or a firewall rule to keep it off your site.

It is also the least documented of the large AI-era crawlers. OpenAI, Anthropic, Perplexity and Meta each publish a page stating what their bots are for, which `robots.txt` tokens they read and where their traffic comes from. We could not find an equivalent page from ByteDance. This guide separates what is attributable from what is reported, then shows how to block Bytespider at each layer and what you can and cannot confirm about a request that carries its name.

## What is Bytespider?

Most of what is known about Bytespider comes from its own request headers and from the companies that see its traffic. Here is each claim with the party that makes it:

| Claim | Who says so | How firm it is |
| --- | --- | --- |
| Bytespider is operated by ByteDance. | The user agent itself names a `bytedance.com` feedback address; Cloudflare’s AI Crawl Control bot reference lists ByteDance as the operator. | Well established. |
| It indexes pages for Toutiao search. | Some logged user agents link to Toutiao’s search webmaster platform; Chinese-language webmaster guides describe it as Toutiao’s search spider. | Plausible, from secondary sources. ByteDance’s platform page did not load any crawler documentation when we checked. |
| It gathers data for training large language models. | Cloudflare’s blog says it is reportedly used for this, including for ByteDance’s Doubao assistant; Vercel’s bot catalog, bots.fyi, files it as an AI crawler tagged for training. | Reported by third parties, not confirmed by ByteDance. |
| It is one of the busiest AI crawlers on the web. | Cloudflare ranked it first among AI crawlers by request volume on its network and the one its customers blocked most often. | Measured by Cloudflare on its own network. |

For a site owner the practical reading is simple. Bytespider belongs to a company you have no agreement with, Cloudflare and Vercel both class it as a crawler rather than an assistant answering a person, and its purpose may include model training. Whether to allow it is a business decision; the rest of this page assumes you want the tools to act on it.

## Bytespider user agent strings

Crawler directories such as Known Agents record several variants. They differ in the device they imitate and the contact they give, but every one carries the same `Bytespider` token:

Bytespider variants as recorded by crawler directories

```http
# Mobile (Android) form
User-Agent: Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)

# Desktop form, feedback address
User-Agent: Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36

# Desktop form, Toutiao webmaster platform link
User-Agent: Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36
```

- **There is no version number** in the token, unlike most crawlers. Match the bare word `bytespider`, case-insensitively.
- **The strings start with `Mozilla/5.0`**, as browsers do. Tools that classify traffic by the opening of the header, or that treat anything Mozilla-shaped as a person, will miscount it.
- **Copying the token is trivial.** Any scraper can send these exact strings, which matters for the verification section below.

## Does Bytespider respect robots.txt?

There is no ByteDance statement we could locate in English, and the published observations point in different directions:

- **Fortune, citing Kasada.** Fortune reported research from the bot-management firm Kasada finding that Bytespider “does not respect robots.txt.” ByteDance and TikTok did not respond to Fortune’s request for comment.
- **HAProxy, on its own site.** HAProxy wrote that most of the AI crawler traffic reaching haproxy.com came from Bytespider, and that some AI crawlers, Bytespider included, don’t identify themselves transparently and ignore instructions in `robots.txt`.
- **Cloudflare.** Its post introducing a one-click AI crawler block ranked Bytespider as the most active and most blocked AI crawler it saw, and noted that `robots.txt` blocks rely on the operator choosing to respect them. That post does not itself say Bytespider ignores the file.
- **Known Agents.** This crawler tracker lists Bytespider as expected to follow `robots.txt`, based on its own measurements.

Those accounts cover different sites and periods, and the behavior may have changed since. The useful answer is the one from your own traffic: add a `robots.txt` rule, wait a few days, then check whether requests naming Bytespider still reach the paths you disallowed. Cloudflare’s AI Crawl Control reports a **Robots.txt violations** count per crawler for your zone, and the log command in the blocking section does the same from raw logs.

## How to block Bytespider in robots.txt

Start with the file, because it costs nothing and any compliant crawler using the token will obey it:

robots.txt: turn Bytespider away from the whole site

```text
User-agent: Bytespider
Disallow: /
```

If your concern is server load rather than access, a narrower rule keeps it away from the URLs that are expensive to render, such as site search and comparison pages, where one crawler can generate an endless supply of unique addresses:

robots.txt: keep Bytespider out of crawl traps only

```text
User-agent: Bytespider
Disallow: /search
Disallow: /compare
Disallow: /cart
```

Then measure whether it listened. This counts Bytespider requests to disallowed paths, grouped by day, so you can see whether the number falls after the rule goes live:

Bytespider requests to a disallowed path, per day

```bash
grep -i "bytespider" /var/log/nginx/access.log \
  | grep -E "\"(GET|HEAD) /(search|compare|cart)" \
  | awk -F"[[/:]" "{print \$2\"-\"\$3\"-\"\$4}" | uniq -c
```

> **Several hosts, several files** Each subdomain answers for itself. A rule in `www.example.com/robots.txt` does nothing for `shop.example.com` or an image host, and Cloudflare found Bytespider leads AI crawlers in how widely it crawls, so put the group on every host you serve.

## Block Bytespider at Cloudflare

An edge block holds regardless of what the crawler does with `robots.txt`. On Cloudflare there are two ways in, and both are available on the free plan:

1. **AI Crawl Control.** Open **AI Crawl Control** for the domain, find Bytespider in the **Crawlers** tab and set its action to **Block**. Cloudflare enforces this by creating or updating a WAF custom rule on the zone, which you can extend with path exceptions.
2. **Your own WAF custom rule.** Write an expression on the user agent, as below, with the **Block** action. This is the route to take if you want the block scoped to certain paths or hosts.
3. **On plans with Bot Management**, match Cloudflare’s detection ID for Bytespider instead of, or as well as, the header text. Cloudflare documents that on the free plan, AI Crawl Control identifies crawlers by user agent, while paid detection goes further.

Cloudflare WAF expressions for Bytespider

```text
# Any plan: match the token, ignoring case
lower(http.user_agent) contains "bytespider"

# Bot Management: Cloudflare's detection ID for Bytespider (from its AI Crawl Control bot reference)
any(cf.bot_management.detection_ids[*] eq 33563853)
```

The [Cloudflare AI Crawl Control guide](/learn/cloudflare-ai-crawl-control) covers the dashboard in detail, including the custom response you can return to a blocked crawler on paid plans.

## Block Bytespider on Vercel, nginx and other servers

- **Vercel, managed ruleset.** The Firewall’s AI Bots ruleset sorts bots into **AI Crawlers** and **AI Assistants**. Vercel’s catalog at bots.fyi places Bytespider under AI Crawlers, so setting that category to **Deny** blocks it. The same switch also denies every other bot in the category, such as GPTBot and ClaudeBot. Choose **Log** first if you want to see the volume before deciding.
- **Vercel, single bot.** For Bytespider alone, add a WAF custom rule on the **User Agent** parameter containing `Bytespider`, with the **Deny** action. Custom rules run before managed rulesets.
- **nginx, Apache, Caddy or a load balancer.** Refuse the request on the token before it reaches your application, so blocked crawls cost almost nothing to serve.

nginx: refuse Bytespider before the app sees it

```text
map $http_user_agent $is_bytespider {
    default        0;
    ~*bytespider   1;
}

server {
    # Keep the file readable so your preference stays visible
    location = /robots.txt { }

    location / {
        if ($is_bytespider) {
            return 403;
        }
        # ... your usual root or proxy_pass
    }
}
```

The separate `/robots.txt` location keeps the file reachable under the block. Crawlers that do read it still see your stated preference, and your violation checks stay meaningful.

## Can you verify Bytespider traffic?

Not to the standard other operators offer. OpenAI, Anthropic and Perplexity publish JSON lists of source addresses; ByteDance publishes no equivalent we could find, and bots.fyi labels Bytespider unverifiable. What remains:

- **Reverse DNS, reported by webmasters.** Secondary sources describe Bytespider hostnames ending in `crawl.bytedance.com`. ByteDance does not document this. If you rely on it, resolve the hostname back to the same address before trusting it.
- **Your CDN’s own detection.** Cloudflare’s paid bot detection uses more than the header. That is a vendor’s judgment rather than ByteDance’s declaration, but it is better than the string alone.
- **Nothing signed.** There is no Web Bot Auth or similar signature to check.

Forward-confirm a reverse DNS name

```bash
ip=203.0.113.7
host=$(dig +short -x "$ip")
echo "$host"                 # expect a name under crawl.bytedance.com
dig +short "$host" | grep -qx "$ip" && echo "forward match" || echo "no match"
```

The asymmetry works in your favor for blocking. A rule on the token stops genuine Bytespider and every impostor using its name, and you lose nothing by stopping both. Verification matters only if you want to allow Bytespider while refusing lookalikes, and that is exactly the case the missing documentation makes hard.

## What blocking Bytespider changes

- **ByteDance crawling stops** at the layer you blocked: completely at the edge, and to the extent it complies at `robots.txt`.
- **Any Toutiao search presence** that depends on the crawler would fade. For most sites outside China this is negligible; check your referrers for Toutiao before deciding.
- **Nothing changes for other assistants.** ChatGPT, Claude, Perplexity, Gemini and Google Search use their own crawlers with their own tokens. A Bytespider block does not touch them.
- **Server load from the crawler disappears.** The public accounts from Kasada, HAProxy and Cloudflare all center on how many requests it sends, so for many sites this is the deciding factor.

If you are tidying up AI crawler access generally, the [AI agent user agents reference](/learn/ai-agent-user-agents) lists the bots you are more likely to want to keep.

## Bytespider and Rebilder

Rebilder’s open-source detector, `@rebilder/agent-detect`, matches the `Bytespider` token and classifies it the same way as GPTBot and ClaudeBot. Here is what happens to a Bytespider request at the gateway:

- **It is classified as an AI agent** on the `bytedance` platform, at medium confidence, because the user agent is the only signal it sends. The pattern follows Cloudflare’s bot reference, which lists `Bytespider` as ByteDance’s AI crawler.
- **It can receive the clean markdown version** of a page wherever one of your sources covers the URL, like other identified AI crawlers. Everywhere else it gets your normal HTML.
- **Your access policy applies to it.** Bytespider sends no signature, so a `deny` or `limit` rule for `unverified` agents, or for every agent (`*`), covers it at the gateway. Keep the `robots.txt` and CDN blocks above if you want it off the site entirely.
- **The Console counts it as AI traffic**, under ByteDance, so you can size it next to the assistants you want to keep.

Blocking it carries no penalty on the readability side. The [free scan](/scan) files Bytespider with the training crawlers, and turning those away while assistants stay allowed leaves the score unchanged.

## What is Bytespider?

Bytespider is a web crawler operated by ByteDance, the owner of TikTok. It identifies itself with the `Bytespider` token in its user agent. Third parties, including Cloudflare, report that it gathers data for ByteDance’s language models; ByteDance has not published an English description of its purpose that we could find.

## Does Bytespider respect robots.txt?

Reports conflict. Fortune, citing research by Kasada, reported that it does not, and HAProxy wrote that some AI crawlers, Bytespider included, ignore `robots.txt` instructions. The crawler tracker Known Agents lists it as expected to comply. Add a rule and check your own logs, or Cloudflare’s Robots.txt violations count, to see what it does on your site.

## How do I block Bytespider?

Add `User-agent: Bytespider` and `Disallow: /` to the `robots.txt` on every host. For a block that does not depend on the crawler’s cooperation, also block the token at your CDN: in Cloudflare’s AI Crawl Control, a WAF custom rule, Vercel’s AI Bots ruleset, or your web server.

## What IP addresses does Bytespider use?

ByteDance does not publish a list we could find. Webmasters report reverse DNS names under `crawl.bytedance.com`, which you can forward-confirm, but that is not documented by ByteDance. For blocking, the user-agent token is enough.

## Will blocking Bytespider hurt my visibility in ChatGPT or Google?

No. Bytespider is ByteDance’s crawler. ChatGPT search reads pages through OAI-SearchBot, Google through Googlebot, and Claude and Perplexity through their own bots. Blocking Bytespider leaves all of them untouched.

## Is Bytespider traffic real or spoofed?

You usually cannot tell from the request. The user agent is easy to copy and ByteDance publishes no verification method. A block on the token stops both the genuine crawler and impostors, which is usually what you want.

## Does Rebilder detect Bytespider?

Yes. The detector matches `Bytespider` as an AI agent on the ByteDance platform, the same way it treats GPTBot and ClaudeBot. The Console counts it as AI traffic, and an access rule for unverified agents covers it. Use `robots.txt` and your CDN to block it from the whole site.

## Related

- [AI agent user agents: the full reference](https://rebilder.com/learn/ai-agent-user-agents)
- [Cloudflare AI Crawl Control explained](https://rebilder.com/learn/cloudflare-ai-crawl-control)
- [meta-externalagent and Meta’s crawlers](https://rebilder.com/learn/meta-externalagent)
- [OAI-SearchBot and ChatGPT search](https://rebilder.com/learn/oai-searchbot)
- [How agent detection works](https://rebilder.com/docs/classification)
- [Which agents Rebilder detects](https://rebilder.com/help/agent-traffic/which-agents-detected)