# [GPTBot in your logs: allow it, block it, or serve it properly](https://rebilder.com/solutions/gptbot)

- **Updated:** 2026-09-10
- **Description:** You found GPTBot in your access log. What it is, how it differs from the agent that fetches on a user’s behalf, and how to decide what to do about it.

Choose how your site responds to OpenAI crawlers and user-directed requests. Understand each role, then set access rules that match your goals.



## What you get

- What the different agents in this family are actually for
- The one distinction that should drive your decision
- How to allow retrieval while declining training, if that is your call
- How to see what you are serving each of them

## These are not all the same agent

The important distinction is between an agent crawling broadly and an agent fetching one URL because a person just asked about it. They arrive with different user agents and they mean completely different things for your business.

| Kind of request | Why it is happening | Request role to evaluate |
| --- | --- | --- |
| Broad crawl | Corpus building, on its own schedule | Training-data collection |
| User-triggered fetch | Somebody asked a question right now | User-directed retrieval |
| Search-style crawl | Indexing for an assistant’s search | Search-related retrieval |

Review crawler access separately from user-directed retrieval. Check the current provider documentation and test how your CDN and bot rules handle each request.

## How to decide

Choose access rules for the uses you want to support. Review training crawlers separately from search and user-directed requests.

- **Publishers whose product is the text itself.** Declining broad training crawls is a defensible commercial position, and plenty of newsrooms have taken it deliberately.
- **Businesses whose product is not the text.** A clinic, a store or a services firm is usually better off being readable: the page exists to be found, and the text is marketing rather than inventory.

> **The decision you can make once** The distinction is expressible in `robots.txt`: the crawling agents and the user-triggered fetcher are separate user-agent strings, so declining corpus building while staying answerable to a live question is a supported configuration rather than a compromise.

## If you allow it, serve it something worth reading

Inspect the actual response for useful business facts. Keep the rendered page clear, and add a compact representation for supported clients where it helps.

middleware.ts

```ts
import { NextResponse } from 'next/server'
import { createGatewayProxy } from '@rebilder/gateway/next'
import { gatewayConfig } from '@/lib/gateway-config'

// Configure storeId and source resolvers in gatewayConfig.
export default createGatewayProxy(gatewayConfig, () => NextResponse.next())
```

Request classification combines headers and a versioned set of detection rules. Verified identity and user-agent matches are labeled separately. Search crawlers receive canonical HTML.

## What is GPTBot?

A crawler user agent used to gather web content. It is distinct from the user agent used when an assistant fetches a page because someone asked a question, and the two have very different consequences if you block them.

## Should I block GPTBot?

Choose the access policy that fits your business. Review training, search and user-directed retrieval separately, using the provider’s current documentation.

## Can I block training but allow live answers?

Yes. The crawling agents and the user-triggered fetcher use different user-agent strings, so `robots.txt` can decline one while allowing the other. That is a supported configuration rather than a workaround.

## How do I see which agents are fetching my site?

Use server-side reporting for direct requests and browser analytics for referred visits. Rebilder classifies observed requests and labels verified identity separately from user-agent matches.

## Related

- [AI agent user agents in your logs](https://rebilder.com/learn/ai-agent-user-agents)
- [ChatGPT traffic](https://rebilder.com/solutions/chatgpt-traffic)
- [Which agents Rebilder detects](https://rebilder.com/help/agent-traffic/which-agents-detected)
- [How agent detection works](https://rebilder.com/docs/classification)
- [The bot roster](https://rebilder.com/bots)