These are not all the same agent
The important distinction is between an agent crawling broadly and an agent fetching one URL because a person just asked about it. They arrive with different user agents and they mean completely different things for your business.
| Kind of request | Why it is happening | What blocking it costs you |
|---|---|---|
| Broad crawl | Corpus building, on its own schedule | Long-term presence in future training |
| User-triggered fetch | Somebody asked a question right now | The answer about you, immediately |
| Search-style crawl | Indexing for an assistant’s search | Retrieval that would have cited you |
Most robots.txt files that block these were written in one line without that distinction being made. The second row is the one worth thinking hardest about: a person is waiting for an answer about you, and blocking it means the answer gets composed without you.
How to decide
There is no universally correct answer here, and anyone whose product depends on you allowing everything will tell you there is. Two honest cases:
- Publishers whose product is the text itself. Declining broad training crawls is a defensible commercial position, and plenty of newsrooms have taken it deliberately.
- Businesses whose product is not the text. A clinic, a store or a services firm is usually better off being readable: the page exists to be found, and the text is marketing rather than inventory.
If you allow it, serve it something worth reading
An allowed fetch that returns a wall of theme markup is barely better than a blocked one: it succeeds, costs the agent most of its budget, and yields little. The same middleware that logs the request can answer it in a fraction of the bytes.
import { gateway } from '@rebilder/gateway'
export default gateway({
source: { kind: 'document', resolve: myPages },
})Classification comes from a fixture corpus of real header samples rather than pattern guesses, and every newly observed agent gets added to it. Search crawlers are classified separately and always receive canonical HTML.