# [GPTBot in your logs: allow it, block it, or serve it properly](https://rebilder.com/solutions/gptbot)

> You found GPTBot in your access log. What it is, how it differs from the agent that fetches on a user’s behalf, and how to decide what to do about it.

- **Updated:** 2026-08-17

GPTBot is one of several user agents in this family, and they do not all do the same job. Blocking the wrong one costs you the thing you probably wanted to keep.



## What you get

- What the different agents in this family are actually for
- The one distinction that should drive your decision
- How to allow retrieval while declining training, if that is your call
- How to see what you are serving each of them

## These are not all the same agent

The important distinction is between an agent crawling broadly and an agent fetching one URL because a person just asked about it. They arrive with different user agents and they mean completely different things for your business.

| Kind of request | Why it is happening | What blocking it costs you |
| --- | --- | --- |
| Broad crawl | Corpus building, on its own schedule | Long-term presence in future training |
| User-triggered fetch | Somebody asked a question right now | The answer about you, immediately |
| Search-style crawl | Indexing for an assistant’s search | Retrieval that would have cited you |

Most `robots.txt` files that block these were written in one line without that distinction being made. The second row is the one worth thinking hardest about: a person is waiting for an answer about you, and blocking it means the answer gets composed without you.

## How to decide

There is no universally correct answer here, and anyone whose product depends on you allowing everything will tell you there is. Two honest cases:

- **Publishers whose product is the text itself.** Declining broad training crawls is a defensible commercial position, and plenty of newsrooms have taken it deliberately.
- **Businesses whose product is not the text.** A clinic, a store or a services firm is usually better off being readable: the page exists to be found, and the text is marketing rather than inventory.

> **The decision you can make once** The distinction is expressible in `robots.txt`: the crawling agents and the user-triggered fetcher are separate user-agent strings, so declining corpus building while staying answerable to a live question is a supported configuration rather than a compromise.

## If you allow it, serve it something worth reading

An allowed fetch that returns a wall of theme markup is barely better than a blocked one: it succeeds, costs the agent most of its budget, and yields little. The same middleware that logs the request can answer it in a fraction of the bytes.

middleware.ts

```ts
import { gateway } from '@rebilder/gateway'

export default gateway({
  source: { kind: 'document', resolve: myPages },
})
```

Classification comes from a fixture corpus of real header samples rather than pattern guesses, and every newly observed agent gets added to it. Search crawlers are classified separately and always receive canonical HTML.

## What this does not do

- It does not tell you to allow everything. Declining broad training crawls is a legitimate position and this page will not argue you out of it.
- It cannot make an assistant mention you. Assistants are not deterministic and no vendor controls their output, so a guaranteed citation is not a thing anyone can sell.
- It does not watch assistants answer questions on your behalf. What it measures is your own server: who asked, what you served, and what you missed.
- It does not verify identity by user agent alone. A header is a claim, which is why signature verification exists separately for agents that support it.

## What is GPTBot?

A crawler user agent used to gather web content. It is distinct from the user agent used when an assistant fetches a page because someone asked a question, and the two have very different consequences if you block them.

## Should I block GPTBot?

It depends on whether your text is your product. Publishers often decline broad training crawls deliberately. For a business whose pages exist to be found, blocking usually costs more than it protects.

## Can I block training but allow live answers?

Yes. The crawling agents and the user-triggered fetcher use different user-agent strings, so `robots.txt` can decline one while allowing the other. That is a supported configuration rather than a workaround.

## How do I see which agents are fetching my site?

From your own request log, classified against a corpus of real header samples. Analytics cannot show you this: agent fetches run no JavaScript and fire no beacon, so they never reach a page-view tool.

## Related

- [AI agent user agents in your logs](https://rebilder.com/learn/ai-agent-user-agents)
- [ChatGPT traffic](https://rebilder.com/solutions/chatgpt-traffic)
- [Which agents Rebilder detects](https://rebilder.com/help/agent-traffic/which-agents-detected)
- [How agent detection works](https://rebilder.com/docs/classification)
- [The bot roster](https://rebilder.com/bots)