RebilderDocs
Documentation menu

Classification

How the gateway decides who is asking — five checks, cheapest first, pure compute — and the guardrail that keeps crawlers on canonical HTML.

The five checks, cheapest first

Detection is pure compute — no network calls, no crypto, a handful of header string scans well under 1ms. Every check that fires is recorded in signals; the highest-precedence one classifies.

  1. Accept: text/markdown (explicit in the list, q-values honored, q=0 excluded) → agent, acceptsMarkdown: true, high confidence. Claude Code and OpenCode send this. Platform attribution comes from Signature-Agent or UA if present, else unknown.
  2. Web Bot Auth headers — Signature-Agent (e.g. "https://chatgpt.com" → chatgpt) → agent, high confidence; a bare Signature + Signature-Input pair → agent/unknown, medium confidence. Parsed only by detect(); verified stays false here — cryptographic verification is the separate async verifyWebBotAuth() step (see below).
  3. Protocol route on the URL pathname — /.well-known/ucp, /.well-known/acp, /mcp, /acp/* (each also matching subpaths) → protocol.
  4. UA heuristics — *fallback only*, capped at medium confidence. Known agent UAs: claude-code, opencode, ChatGPT-User, OAI-SearchBot, GPTBot, PerplexityBot, Perplexity-User, Gemini/Google-Extended, anthropic-ai/claude-web (plus ClaudeBot/Claude-User). Crawler UAs: Googlebot → crawler/googlebot, bingbot → crawler/bingbot.
  5. Default → human. High confidence when a browserish Accept (text/html) or UA (Mozilla/) is present; low otherwise (empty or garbage headers).

From detection to serving path

The gateway maps the detection onto a serving path (a Response means markdown was served; null means your HTML pipeline runs as if the gateway weren’t there):

DetectionPathBehavior
Agent with Accept: text/markdown or an identified platformmarkdownRender from sources; Response returned
Agent (signed/identified) on a protocol route, no markdown AcceptprotocolRouted to the protocols hook — signed protocol clients must reach the protocol handler
Agent, unidentified, no markdown Accept (e.g. bare Signature pair)htmlnull — never guess a format nobody asked for
Humanhtmlnull — your HTML pipeline runs as if we weren’t there
Crawler (Googlebot, bingbot)htmlnull, always — see the guardrail below
Protocol routeprotocolconfig.protocols hook when wired (see Protocols); null otherwise — the event records the demand either way

The Googlebot guardrail

This is the cloaking guardrail in practice: UA sniffing alone never triggers substantive differences — it only ever selects a format transformation of the same substance. Serving a crawler markdown it didn’t negotiate for would look like cloaking; refusing to is a hard rule, not a setting.

Confidence semantics

DetectionResult
const result = detect({ headers: req.headers, url: req.url, method: req.method })
// {
//   kind: 'agent' | 'human' | 'crawler' | 'protocol',
//   platform: 'claude-code' | 'opencode' | 'claude' | 'chatgpt' | 'gemini'
//           | 'perplexity' | 'googlebot' | 'bingbot' | 'unknown' | null,
//   verified: boolean,          // ALWAYS false from detect() — verification is the
//                               // separate async verifyWebBotAuth() step
//   acceptsMarkdown: boolean,   // Accept explicitly lists text/markdown with q > 0
//   signals: string[],          // ordered checks that fired, e.g. ['accept:text/markdown', 'ua:claude-code']
//   confidence: 'high' | 'medium' | 'low',
// }
  • high — an explicit negotiation signal: Accept: text/markdown, or an identified Signature-Agent. Also the browserish-human default.
  • medium — UA heuristics and bare signature pairs. UA-derived classifications are *capped* at medium: UA strings are trivially spoofable, so they may only ever select a format transformation of the same substance, never anything substantive.
  • low — the default human classification with empty or garbage headers.
  • Anything substantive — preference-payload personalization and protocol transactions in particular — requires verified: true from the separate cryptographic step (verifyWebBotAuth()), never a parsed claim. Do not gate anything security-sensitive on an unverified platform claim.

Fixture-driven testing

Detection is tested against a corpus of real observed header samples — one fixture file per agent, each carrying raw headers plus the expected classification. The test suite loads every fixture and asserts every sample, so adding a fixture *is* adding a regression test. Every newly observed agent (new headers, new signatures, new UA strings) gets captured there first, then the detector is taught to match. The corpus grows with live traffic and is treated as data, not test scaffolding.

Verification (Web Bot Auth)

Verification is a separate async step from detection: detect() stays synchronous, parse-only, and its verified is always false. verifyWebBotAuth() cryptographically verifies a Web Bot Auth signature chain (RFC 9421 Ed25519 message signatures) against a caller-injected key registry — pure crypto over ambient WebCrypto, zero network, and it never throws: every malformed or adversarial header resolves to { verified: false, reason }.