AI agent user agents: a field guide to every string in your logs

How to read AI agent user agents

The strings sort into four families, and the family matters more than the version number. It tells you why the request happened and how much intent sits behind it:

• User-initiated fetchers — sent while a human is mid-conversation with an assistant: ChatGPT-User, Perplexity-User. One fetch maps to one real person asking about your product right now.
• Index and training crawlers — bulk fetches building a corpus: GPTBot, OAI-SearchBot, PerplexityBot. No live user attached, but they decide whether the assistant knows your store exists.
• Terminal agents — CLI coding agents that fetch pages during a working session and ask for markdown outright: claude-code, opencode.
• Search crawlers — Googlebot, bingbot. Recognized by the same detector, but classified as crawlers, not agents — and always served canonical HTML.

OpenAI: ChatGPT-User, GPTBot, and OAI-SearchBot

Merchants searching for the ChatGPT shopping user agent are almost always looking for ChatGPT-User — the string that appears when a shopper asks ChatGPT about a product and the assistant fetches your page mid-conversation. The GPTBot user agent is a different animal: OpenAI’s bulk crawler, fetching broadly with Accept: */* and no live user attached. Three strings, three jobs:

Why the first row matters most: OpenAI has shifted toward discover in chat, transact on the merchant’s site — the assistant shortlists, then the human lands on your store to buy. A ChatGPT-User fetch is the discover step happening in real time. Serve it a parseable page and you are in the shortlist; serve it 90KB of theme markup and you are trusting a parser to dig.

Anthropic: the Claude user agent family

The Claude family member our corpus holds full captures for is claude-code, Anthropic’s terminal coding agent — and the most explicit requester in that corpus, because it does not make you guess: it sends an Accept: text/markdown media range alongside its CLI user agent. A captured request, verbatim:

GET /products/olive-oil-500ml HTTP/1.1
Host: store.example.com
User-Agent: claude-code/1.0.58 (external, cli)
Accept: text/markdown;q=1.0, text/html;q=0.8, text/plain;q=0.5, */*;q=0.1
Accept-Encoding: gzip, deflate, br

Honest corpus note: the last row is matched by our detector, but we do not yet hold full raw-header captures for those markers. When one lands in live traffic it becomes a fixture first — headers, URL, expected classification — and this page gets updated.

Perplexity: PerplexityBot and Perplexity-User

The Perplexity user agent comes in two forms, and the split mirrors OpenAI’s: an index crawler that builds the corpus behind its answers, and a user-initiated fetcher that fires when a person asks something your page answers. Both classify as agent traffic on the perplexity platform:

Two practitioner details from the captures. First, both send fully browser-like Accept: text/html,... headers — nothing but the UA separates them from a human visitor, which is why UA matching sits last in the detection order. Second, note the parenthesis placement: unlike ChatGPT-User, the compatible; ... clause sits inside the (KHTML, like Gecko; ...) parenthetical. Regexes built on the ChatGPT shape silently miss Perplexity. Match tokens, not shapes.

OpenCode, Googlebot, and bingbot

The rest of the corpus: a second markdown-requesting terminal agent, and the search crawlers every store must keep on canonical HTML.

Headers beat user agents: Accept and Web Bot Auth

Accept: text/markdown is ordinary HTTP content negotiation, and it outranks every UA check for one reason: it does not claim an identity, it states a want — and the want is the only thing worth acting on. Honor q-values (q=0 means excluded) and treat an explicit text/markdown listing as a high-confidence agent signal, whatever the UA says.

The second header family is Web Bot Auth: three headers that let an agent platform cryptographically sign its requests. Here is a captured ChatGPT-User/2.0 request carrying all three:

GET /products/espresso-grinder HTTP/1.1
Host: store.example.com
User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/2.0; +https://openai.com/bot
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Signature-Agent: "https://chatgpt.com"
Signature-Input: sig1=("@authority" "signature-agent");created=1754265600;expires=1754266200;keyid="JrQLj5C_-uusgQwmSFvHYK2PxYlbcO9zRMoDHIQdVRo";tag="web-bot-auth"
Signature: sig1=:TWFrZSBhZ2VudHMgZmlyc3QtY2xhc3MgY2l0aXplbnMgb2YgdGhlIHdlYi4=:

• Signature-Agent — a quoted origin naming who signs ("https://chatgpt.com"). Present → platform attribution at high confidence.
• Signature-Input — which request components are covered, plus metadata: created, expires, a keyid, and the tag="web-bot-auth" label.
• Signature — the signature bytes themselves. A bare Signature + Signature-Input pair without Signature-Agent still means agent — unknown platform, medium confidence.

Accurate retrieval: content negotiation vs llms.txt

~4.2x

300k-domain study, mid-2026

Payload reduction when markdown is served

98.6%

Deterministic before/after capture, same product page

Conversion lift, AI-referred visitors vs search

~38%

Cyber Week 2025 retail data

How to detect AI agents properly

Detection order matters more than detection coverage. One guardrail runs before everything, then the cheap, strong signals, then the weak ones — this is the exact classification order our gateway ships, pure compute, well under 1ms:

1. Known crawler UA? Classify crawler and stop. The guardrail runs before everything: a Googlebot or bingbot token always classifies as crawler — canonical HTML — even when every other signal fires.
2. Then check Accept: text/markdown. Explicitly listed with q > 0 → an agent asked for machine-readable output. High confidence, and the only check that carries intent rather than identity.
3. Then Web Bot Auth headers. Signature-Agent attributes the platform at high confidence; a bare Signature pair means agent, unknown platform, medium.
4. Then protocol routes. Requests to /.well-known/ucp, /.well-known/acp, /mcp, or /acp/* are protocol probes — worth logging today, even though our protocol endpoints are announced rather than live.
5. UA heuristics last, capped at medium confidence. Every string on this page, matched as a fallback. A UA match may select a format transformation of the same substance — never different prices, claims, or availability.
6. Default: human. A browserish Accept: text/html or Mozilla/ UA → high confidence; empty or garbage headers → low.

You can implement this order yourself — headers first, UA last, crawlers pinned to canonical HTML — or install it: Rebilder’s gateway ships this exact classifier free, with adapters for Next.js, Shopify, Express, Fastify, and Cloudflare Workers, and answers Accept: text/markdown with markdown rendered from your own product data. The full ordering is documented in Classification.

A living corpus, not a static list

Every user agent on this page exists because a fixture exists: a captured sample carrying the raw headers, the URL, and the expected classification. The detection test suite loads every fixture and asserts every sample, so adding a newly observed agent to the corpus *is* adding a regression test. When a new string or signature shows up in live traffic, it gets captured first, matched second, and published here third. The updated date at the top of this page tracks the corpus, not copyedits.

The fastest way to see which of these strings are already hitting your store: the free Console records every classified request — agent, human, crawler, or protocol probe, which platform, and what was served — and its side-by-side preview shows exactly what each requester receives.