Status: DRAFT. Published for implementation and criticism. Scores produced under 0.1 are declared non-comparable to a future 1.0 unless 1.0 is calibrated first (§12).
| Standard | The Agent Readability Spec (ARS), version 0.1.0 |
| Metric | The Agent Readability Score — an integer 0–100 and a band, or one of three non-grades |
| Reference implementation | @rebilder/agent-readability — scorer at ., probe at ./probe |
Ruleset hash (ARS 0.1.0) | 51176d88c6ca1478a9d3510c09eb8eb8181cff86beaea16b96812eb25c5910e0 |
| Corpus | packages/agent-readability/conformance/ — 38 fixture directories, not frozen (§2.4.3) |
| Editors | Rebilder |
0. Disclosures, at the top
0.1 Conflict of interest
ARS is authored by Rebilder, which sells content-negotiation middleware. Two ARS checks (D2.1 content negotiation, D5 structured data) can be satisfied by Rebilder's free gateway, and reaching an A requires both. The full ruleset, thresholds, and conformance corpus are published so this bias is auditable rather than hidden.
This notice MUST be printed at the top of any rendering of this document, on every published scan result, and in the --help output of any conformant command-line tool.
The claim in the second sentence is not rhetorical and is not asked to be taken on trust: it is arithmetic over the published weights, and §8 derives it. The editors do not promise to move this standard to a neutral domain if a second implementation appears. That is a promise conditioned on a trigger the editors cannot cause, and a conditional promise on an uncontrollable trigger is not a commitment.
0.2 The measurement notice
The following two lines MUST appear, verbatim and unabridged, on every surface that displays an ARS grade or score — web page, badge, API response documentation, terminal output, PDF, or slide:
ARS measures format and retrievability. It does not measure whether the facts are true, or whether any assistant cites this page.
An implementation that displays a grade without this notice is not conformant (§10.5).
1. Scope
1.1 Purpose
ARS scores a single fetched HTTP response set for one URL and answers one question:
Are the facts a caller needs for this kind of page present, findable, and cheap relative to what that kind of page costs?
The unit of measurement is one URL. Site-level aggregation exists in the reference implementation as an informative convenience and is not normative in 0.1; a sampling rule MUST NOT be promoted to normative before it has survived a version.
1.2 What ARS does not measure
ARS does not measure, and no conformant implementation may claim that it measures:
- Truth. ARS never evaluates whether a stated price, hour, claim, or date is correct. A page that states four fabricated facts in perfect JSON-LD scores exactly as a page that states four true ones.
- Citation, ranking, or inclusion in any assistant's answer. ARS has no visibility into any model's retrieval or ranking. A high ARS is not a prediction that ChatGPT, Claude, Perplexity, Gemini, or any other system will read, cite, or recommend the page.
- Cloaking compliance. ARS identifies itself honestly on every request (§3.2). It therefore cannot detect content that is varied by
User-Agent, including content varied specifically forrebilder-ars. ARS is not a cloaking-compliance certification. - Content quality, accuracy, legality, accessibility, or performance. ARS is not a WCAG audit, a Core Web Vitals measurement, or an editorial judgement.
- Commercial merit. A price is scored as present or absent, never as high or low. ARS contains no cross-site price comparison and MUST NOT be extended into one.
- Whether a site should allow agents. A deliberate opt-out is a non-grade, not a failure (§7.3).
- JavaScript-rendered output. ARS never executes scripts. D1.3 estimates render independence from the served HTML and is labelled
heuristicfor exactly that reason (§5.3.3).
1.3 Notational conventions
The key words MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL in this document are to be interpreted as described in BCP 14 (RFC 2119, RFC 8174) when, and only when, they appear in all capitals.
All arithmetic in this document is integer arithmetic unless stated otherwise. ⌊a ÷ b⌋ denotes exact integer division of non-negative integers (§6.1). Byte counts are decoded UTF-8 bytes (§3.4). KiB is 1024 bytes; MiB is 1048576 bytes.
1.4 Artifacts a conformant implementation reads
| Artifact | Role |
|---|---|
| This document | The prose definition |
conformance/ | The normative test of agreement. JSON in, JSON out, no TypeScript required (§10) |
DEFAULT_RULESET | Every constant, in one hashed object (Appendix A) |
Where this document and the conformance corpus disagree, the corpus is authoritative and the disagreement is a defect in this document; report it. Where the corpus and the reference implementation disagree, the corpus is authoritative and the disagreement is a defect in the implementation.
1.5 Terminology
Evidence bundle — the complete, self-contained input to scoring: the captures, the request headers used, the vantage, and the target. Type ArsEvidence.
Agent probe / agent representation — the response to the request whose Accept header prefers a machine representation, and the decoded body of that response. This is what a caller receives, and it is what D3 and D4 score.
Browser control probe / HTML representation — the response to the request whose Accept header prefers HTML.
Structural representation — the representation that carries a tokenizable HTML document: the browser representation when it has one, otherwise the agent representation when it has one, otherwise none. D1.3 and D5 score this.
Core fact / extended fact — the facts a page kind is expected to state (§4.2). Core facts carry full weight; extended facts carry ¼ credit, and ¼ is the only extended-credit figure in this standard.
Corroborated — the same normalised value for the same fact kind was observed from two or more distinct sources, counting across both representations (§4.6.4).
Measured / heuristic — the basis of a check. A measured check reports a property of the bytes and headers that were received. A heuristic check reports an inference about them. Every check declares its basis in every result, and a heuristic value MUST NOT be rendered without its label.
Vantage — public (a third party, e.g. the public scanner), authenticated (a signed-in operator of the origin), or self (a run by or for the origin's own operator, including a local CLI run). Only public is gated on the scanner token (§7.5).
2. Versioning, determinism, and identity
2.1 Version scheme
Versions are ARS <MAJOR>.<MINOR>.<PATCH>.
| Component | Meaning | Score comparability |
|---|---|---|
| PATCH | No expected.json in the conformance corpus changes. Wording, documentation, performance, refactors. | Scores comparable |
| MINOR | Scores may change. Any ruleset constant, any extraction rule, any classification table, any tokenizer recovery rule. | Scores NOT comparable across a MINOR. Every published surface re-scores in a batch |
| MAJOR | The dimension set or the 100-point allocation changes. | Not comparable |
2.2 The PATCH gate
A PATCH release whose golden files differ MUST fail the build. This is a CI gate in the reference implementation, not a convention.
The consequence is deliberate and is the point of the rule: a bug fix in the tokenizer or the extractor that changes any fixture's expected.json cannot ship as a PATCH. It is a MINOR, with everything that follows — re-scoring, an incomparability notice, no silent drift. Implementations that publish scores MUST adopt an equivalent gate, or MUST NOT claim PATCH-level comparability.
2.3 The determinism guarantee
For a given
evidenceHash,rulesetHashandcorpusHash,score()returns a byte-identicalArsResultin any conformant implementation, in any language.
A number without all three hashes is not an ARS score. A surface that displays a grade MUST also make the three hashes retrievable for that grade. A badge, which has no room for them, MUST display the specVersion and a rulesetHash prefix, because a pinned-token update changes rulesetHash at an unchanged specVersion and two incomparable scores must not render identically.
Three properties are required to hold this guarantee, and each cost something to obtain:
- No floating point in the scoring path (§6.1). Two implementations that disagree in the third decimal disagree about grades at a band boundary, and the boundaries are what gets published.
- No third-party HTML parser. A parser version is an unversioned input to a versioned score: a dependency upgrade that changes how a malformed
<divis recovered changes scores with no ARS version bump, no conformance diff, and nothing in the release notes. A conformant implementation MUST implement the tokenizer rules itself, or MUST pin its parser and treat a parser upgrade as a MINOR. - No clock, no network, no randomness, and no language model inside
score(). All time-varying and network-varying inputs live in the evidence bundle, which is hashed.
2.4 The three hashes
2.4.1 rulesetHash
SHA-256, lowercase hex, of the RFC 8785 (JCS) canonical JSON serialisation of the frozen ruleset object. For ARS 0.1.0 this value is:
51176d88c6ca1478a9d3510c09eb8eb8181cff86beaea16b96812eb25c5910e0
An implementation that produces a different value for the ARS 0.1.0 ruleset has a serialisation defect, a constant transcription error, or both, and MUST NOT publish rulesetHash as if it agreed.
2.4.2 evidenceHash
SHA-256 of the JCS canonical JSON of the evidence bundle with volatile fields removed:
capturedAt(bundle-level wall clock) is excluded.- The response headers
date,age,set-cookie,x-request-id,cf-ray,report-toare excluded, matched case-insensitively; all remaining header names are lowercased before hashing. - Response bodies are included in the hash where present.
bodySha256is already carried in each capture, but hashing the bytes as well means a bundle whose body was edited without updating its digest does not silently keep its identity.
Without the exclusion list the hash changes on every scan of every real site — Date alone guarantees it — and replay, which is the only reason the hash is published, becomes impossible.
2.4.3 corpusHash
SHA-256 over the frozen conformance corpus. In ARS 0.1 this value is not yet a corpus digest, and the standard says so rather than implying a freeze it does not have: the corpus is still open, fixtures are still landing, and the constant shipped is the SHA-256 of the ASCII string ars-0.1-corpus-unfrozen:
22e436d6f50e1804d72bb45e8e5cd9c00a2d4b0e4e924dc6573911c065a6dfdb
Anyone who computes a digest of the shipped corpus will find it does not match. That is the correct signal: a score carrying this constant is a draft score. At corpus freeze the constant is replaced by the real digest, which is a MINOR.
2.5 What rulesetHash does not cover in 0.1
Stated here rather than discovered later. Three tables are inputs to a score and are not members of the hashed ruleset object, because the published ArsRuleset type has no field for them:
SUBPOINTS— the point split inside each check that scores a conjunction of named conditions (§5.3–§5.8). Published, exported, and reproduced in this document.- The classification tables — schema.org type → page kind,
og:type→ page kind, and the URL path grammar (§4.5). - The extraction tables — the label lexicon, availability vocabulary, no-published-price phrases, and schema.org property aliases.
All three are pinned data, not inference, and all three are pinned in practice by the conformance corpus: an edit to any of them moves an expected.json and is therefore at minimum a MINOR (§2.1). Folding them into the hashed object is an open item for 0.2 (Appendix D).
2.6 Ordinality
ARS is ordinal, not cardinal. Bands headline; the integer is secondary; there are no decimals, ever. A one-point difference between two sites is not a finding, and an implementation MUST NOT present it as one — no ranked lists keyed on the integer, no "you beat them by 2".
3. The probe
The probe is the impure half. It is not part of score(), and everything it decides that could change a score is pinned in the ruleset for exactly that reason: two implementations with different caps produce different scores, so the caps are inputs to the score and therefore inputs to the hash.
3.1 The request set
Per URL, two sequential requests; per origin, three requests whose results are cached for at least one hour (24 hours in the public scanner). A cold single-URL scan is therefore 5 requests and a warm origin is 2. A third request to the target URL is made only when a substance divergence is detected (§5.4.4).
| # | Request | Accept | Purpose |
|---|---|---|---|
| 1 | GET <origin>/robots.txt — fetched first | text/plain, */*;q=0.5 | Policy. Nothing else is fetched until this resolves |
| 2 | GET <url> — agent probe | text/markdown;q=1.0, text/html;q=0.8, text/plain;q=0.5, */*;q=0.1 | What a caller that prefers a machine representation receives |
| 3 | GET <url> — browser control | text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 | What a caller that prefers HTML receives |
| 4 | GET <origin>/llms.txt | text/plain, */*;q=0.5 | D6.3 |
| 5 | GET <origin>/.well-known/ucp | text/plain, */*;q=0.5 | D6.5 |
| (6) | GET <url> — parity confirmation, ≥30 s after #2, only on detected divergence | same as #2 | §5.4.4 |
Every request carries the same User-Agent:
rebilder-ars/0.1 (+https://rebilder.com/bots)
3.2 The two probes differ only in Accept — and what that means we cannot see
Normative: requests #2 and #3 MUST be identical in every respect except the Accept header. Same User-Agent, same cookies (none), same protocol, same everything else.
This is what makes the parity check (D2.4) meaningful, and it is what prevents ARS from penalising a correct content-negotiation install, which negotiates on Accept and not on identity.
We imitate agent intent, never agent identity. ARS never sends a User-Agent belonging to Googlebot, ChatGPT, Claude, or any other operator's crawler. Impersonating another party's crawler to test a site is a thing we will not do, and every request we make is honestly attributable to us.
The printed consequence, stated here next to the parity check rather than in a footnote:
ARS cannot detect
User-Agent-targeted cloaking, including cloaking targeted atrebilder-arsitself. Both of our probes identify honestly, so a site that serves identical fabricated substance to both scores as though it were truthful. ARS is not a cloaking-compliance certification.
D2.4 detects only the narrow case where the same identity receives different substance under different Accept headers. That is the case a content-negotiation product creates, so it is the case this standard is obliged to check.
3.3 Body cap
Normative: 2 MiB (2 097 152) decoded bytes.
- The body MUST be read as a stream and aborted at the cap. Reading the whole body and truncating afterwards is forbidden: the point of the cap is that the bytes are never all in memory.
- A capture that hit the cap MUST record
truncated: true. - A truncated capture produces the non-grade
unscored / truncated-evidence(§6.8). It is not scored, because every quantity would be wrong in the site's favour: the fact set would be whatever fell inside the first 2 MiB, the context cost would be measured against a byte count that is not the page's, and the token estimate would understate by however much was never read. Silently scoring a truncated body makes the worst sites score best on the one number the product is built on. - Where a truncated byte count is displayed at all it MUST be rendered as
≥ N bytes, never as a measurement.
3.4 Charset, bytes, and offsets
- Bodies MUST be decoded according to the
charsetparameter of the responseContent-Type, defaulting to UTF-8. ArsHttpCapture.bytesis the decoded UTF-8 byte length, not the transfer length and not the JavaScript/Java/Go string length. This is specified so that an implementation in another language reproduces our offsets exactly.- Every fact offset in a result is a UTF-8 byte offset into the decoded body. On a page with any multibyte character ahead of the price — a
€, an em dash, a CJK product name — a UTF-16 index and a UTF-8 offset differ, and a D4.1 band boundary sits between them. Conformance fixture048-shift-jis-charsetexists to make this checkable: the same JSON-LD block sits at UTF-8 offset 344 and at Shift_JIS offset 310. transferBytesis deliberately not a field. Runtimes auto-decompress and platforms manageAccept-Encoding, so a hashed field that will be absent or platform-dependent is worse than no field.
3.5 Redirects
Normative: at most 3 hops. A response chain longer than the limit produces unscored / too-many-redirects. Every hop MUST re-resolve DNS and re-validate the destination against the implementation's SSRF policy; re-parsing the URL without re-resolving is not sufficient.
The limit is inclusive: three hops is scored, four is not. Conformance fixture 042-redirect-chain-3 pins the boundary.
3.6 robots.txt
robots.txtMUST be fetched first, before the target URL.- Parsing follows RFC 9309 in the subset ARS reads:
User-agent,Allow,Disallow,Sitemap.Crawl-delayis honoured by the probe and never scored — waiting longer is not worse markup.HostandClean-paramare ignored without being counted as malformed. Any otherfield: valueline, and any line that is not blank, a comment, or afield: valuepair, incrementsmalformedLines. - Adjacent
User-agent:lines accumulate into one group; the first rule line closes the group to further agents (RFC 9309 §2.2.1). A rule that appears before anyUser-agent:line incrementsorphanRulesand belongs to no group. - Group selection is most-specific-match,
*as fallback, case-insensitive. Within the selected group, the longest matching path pattern wins and a tie goes toAllow(RFC 9309 §2.2.2). - Path patterns support
*and a trailing$. Implementations MUST NOT compile an attacker-supplied pattern into a regular expression;robots.txtis untrusted input andnew RegExp(patternFromTheInternet)is a denial-of-service primitive. - A persistent 5xx — one retry after 5 s, then still 5xx — produces
unscored / robots-unavailable. A single hiccup MUST NOT un-score a domain. A 4xx (including 404) ismissing, not an error, and is scored: an absentrobots.txtdisallows nothing.
3.7 What is not probed
ARS probes /robots.txt, /llms.txt and /.well-known/ucp, and nothing else. There is no guessing at /sitemap.xml, /mcp, /acp, /openapi.json, or any other path.
Credit for a machine endpoint (D6.5) comes from declaration — a Link header, a <link rel="alternate">, an entry in llms.txt, or a Sitemap: line in robots.txt. The editors' stated principle:
An endpoint an agent cannot find is an endpoint that does not exist.
The three probed paths are the disclosed exception: they are published conventions with published locations, and an agent looking for them knows where to look.
3.8 The evidence bundle
The evidence bundle is the only input to scoring besides the ruleset. It carries evidenceVersion, the target URL and origin, the six probe records, the vantage, and capturedAt.
Each probe record carries the request headers actually sent and either a capture or a typed error (timeout, unreachable, non-2xx, blocked-redirect, too-many-redirects, challenge, policy-rejected).
Each capture carries requestedUrl, finalUrl, the redirect chain, status, headers as Record<string, string[]>, bytes, bodySha256, truncated, and OPTIONALLY body.
- Headers are multi-valued by type, not by convention.
Linkis routinely sent several times and is scored; an implementation that collapses repeated headers to the first value loses declared alternates and machine endpoints. Fixture051-multi-value-headerspins this, includingLinkvalues that contain commas inside<…>, which MUST NOT be split on the comma. bodyis optional because a published evidence bundle carries the hash and not the bytes. A bundle without bodies cannot be re-scored, and a conformant scorer MUST returnunscored / evidence-incompleterather than inventing a zero.
3.9 Probe conduct
These requirements bind any implementation that fetches third-party origins. They are not inputs to the score, but a standard that instructs people to fetch arbitrary URLs owes them:
- SSRF. Resolve the hostname; reject if any returned A/AAAA record is private, reserved, link-local, CGNAT (
100.64.0.0/10), benchmarking (198.18.0.0/15), multicast, or240.0.0.0/4; connect pinned to the validated IP withHostand SNI preserved; re-resolve and re-pin on every redirect hop. Reject.internal,.local, and single-label hosts. Handle IPv4-mapped IPv6. - Local mode is opt-in per invocation. An entry point that permits private hosts or plaintext HTTP MUST be a separate module requiring an explicit per-invocation flag, and MUST NOT be reachable from an HTTP transport. A boolean on a shared config object will eventually be passed as
trueby a hosted endpoint. - Politeness. At most one concurrent probe per host, a small global concurrency limit, at least one second between probes to the same host, and a soft per-run cap with an explicit override. The limiter MUST be injected rather than module-global: module state is correct in a CLI process and wrong in a warm serverless instance, where it would begin rejecting unrelated callers after a fixed lifetime count.
- Robots. Anything that produces a published or permanent artifact obeys
robots.txt,Crawl-delay,429/Retry-After, and any published opt-out endpoint.
4. Page kinds, facts, and extraction
4.1 Universality: fixed weights, varying fact schema and byte reference
Per-vertical weights are the wrong design. They make an 88 mean different things on different pages and they invite site-type shopping. ARS therefore uses one 100-point allocation for every page.
What varies by page kind is what counts as a fact and what a page of that kind reasonably costs in bytes. Neither is a weight.
ARS does not measure how much you say. It measures whether the facts a caller needs for this kind of page are present, findable, and cheap relative to what that kind of page costs.
The byte reference is specifically what stops long-form content — articles, documentation, datasets — from being structurally capped around C, which is what every design that used absolute byte bands did. §11.5 walks a 14 KB magazine feature that outscores a 991-byte product page, and states honestly where the mechanism stops working.
4.2 The eight page kinds
The set is closed at eight. Every kind has a complete profile; there is no "no profile" path. The mapping MUST be total, and every fact name MUST be a member of the fact vocabulary.
| Kind | Core facts | Extended facts (¼ credit) | Byte reference |
|---|---|---|---|
product | title, price, currency, availability | brand, sku, shipping, returns, description | 8 KiB (8192) |
collection | title, item-count, item-link | price, availability, description | 16 KiB (16384) |
article | title, author, published | updated, section, description | 40 KiB (40960) |
place | org-name, address, hours, phone | email, service-area, primary-action-url, description | 8 KiB (8192) |
service | title, price, primary-action-url | duration, eligibility, service-area, description, updated | 8 KiB (8192) |
faq | title, question-answer | updated, description | 16 KiB (16384) |
document | title, updated, authority | description, primary-action-url, section | 24 KiB (24576) |
unknown | title, description, primary-action-url | updated, org-name | 16 KiB (16384) |
The complete fact vocabulary is: title, description, updated, published, price, currency, availability, brand, sku, shipping, returns, org-name, address, hours, phone, email, service-area, author, section, authority, primary-action-url, eligibility, duration, question-answer, item-count, item-link.
4.3 Definitions of the fact terms
These five terms are defined normatively because a score that depends on them is not reproducible otherwise.
primary-action-url — a same-origin-or-declared URL reachable from an <a href>, a <form action>, or a JSON-LD potentialAction.target, whose anchor text or name matches the ruleset's action lexicon. The lexicon is pinned (Appendix C) and MUST NOT be extended per page, per site, or by inference; primary-action-url is a core fact for service and unknown, so a lexicon that grew at runtime would take the whole score with it.
authority — an organisation or person named as the publisher of the content, extracted from JSON-LD publisher / author, <meta name="author">, or a rel="publisher" link.
org-name — JSON-LD Organization.name / LocalBusiness.name, og:site_name, or <meta itemprop="name"> on an org-typed scope.
item-count / item-link — for collection: the number of distinct item links inside the largest repeated-structure block, and whether those links resolve to same-origin URLs. Counting from the largest repeated block specifically is what stops a navigation menu being read as a product grid.
question-answer — a FAQPage/Question JSON-LD pair, or a <dt>/<dd> pair, a <details><summary>, or a heading followed by a paragraph. Counted once as present, never per item. A page with 40 questions is not four times better than a page with 10; fixture 017-faq pins this.
price is satisfied by an explicit "no published price" signal as well as by a number. ARS scores honesty, not disclosure: a service page that says "pricing on request" has told the agent what it needs to know. Fixture 013-service-pricing pins it — and also exposes an unresolved inconsistency with the D5.2 required-property table, recorded in Appendix D.
4.4 unknown is not an easy exit
unknown carries the smallest core set, but it still scores D5.2 against "any recognised schema.org type is present", which requires having a type at all. And because extraction runs against the resolved profile, a page that conceals its kind does not merely fail to be credited for its commerce facts — those facts are absent from the published fact set entirely.
Hiding your page kind costs up to 7 points on D5.2, costs coverage on D3.1, and buys nothing. Fixture 015-unknown-untyped states the arithmetic on a page that ships a clean JSON-LD block under a private @context: D5.1 earns 5/5, D5.2 earns 0/7, and the page's stated price and availability do not appear in facts at all.
pageKind and pageKindConfidence are reported on every result, including non-grades.
4.5 Classification: two passes
Fact extraction needs a profile; the profile comes from the page kind; the page kind looks like something one would infer from the facts. That circularity is cut with a hard wall:
Pass 1 extracts a fixed, kind-independent probe set: JSON-LD @type, microdata itemtype, RDFa typeof, og:type, URL path segments, presence of a parseable currency token, day-name/HH:MM adjacency, and presence of a telephone. Only pass-1 output may inform classification.
Pass 2 extracts against the resolved profile.
Precedence, applied in order:
| # | Signal | Confidence |
|---|---|---|
| 1 | JSON-LD @type, through the pinned type table | high |
| 2 | microdata itemtype, then RDFa typeof, through the same table | high |
| 3 | og:type | medium |
| 4 | URL path grammar | medium |
| 5 | Structural: opening-hours shape and a telephone → place | medium |
| — | anything less | low → unknown |
Ties within one signal are broken by document order, then lexicographically by type name. Below medium confidence the kind is unknown; an implementation MUST NOT guess into a vertical. Guessing product on an article scores the article against a fact set it had no reason to carry, and the publisher cannot argue with a number.
pageKindBasis is always heuristic in 0.1.
4.6 Facts, sources, and corroboration
4.6.1 Sources
A fact observation carries one of five sources, in this precedence order: json-ld, microdata, meta, html-text, negotiated. Precedence is used only to break an offset tie.
4.6.2 Which representation is extracted
- D3 (coverage) and D4 (position) score the agent representation. ARS measures what a caller actually receives. A merchant whose Markdown omits
availabilityloses coverage even though the HTML declares it, because the caller never saw it. - D1.3 (render independence) and D5 (structured data) score the structural representation, because those checks are about the document a caller that only reads HTML gets. JSON-LD does not live in Markdown, and "would this work without JavaScript" is a question about HTML.
4.6.3 Presence facts
Some facts are compared by value (price, title, phone). Others are only ever "declared or not": shipping, returns, hours, service-area, eligibility, question-answer, item-link. These normalise to the literal string declared from every source, deliberately — two sources that word the same shipping policy differently must still corroborate one another, and a free-text comparison of policy prose would not be deterministic.
4.6.4 Merging and corroboration
For each fact kind observed in the agent representation:
- The representative observation is the one with the lowest byte offset; ties are broken by source precedence, then lexicographically by normalised value. Lowest offset, because D4.1 scores the offset at which a fact first becomes available.
corroboratedis true when two or more distinct sources, counted across both representations, reported the representative's exact normalised value. A source that reported a different value corroborates nothing.- A fact observed only in the HTML does not enter the scored fact set. Corroboration strengthens a fact the agent received; it never conjures one the agent did not.
The published fact list is sorted by offset, then by fact kind.
4.7 Text extraction (normative)
Counted text is every text node except:
- the contents of
<script>,<style>and<template>; - any element carrying the
hiddenattribute, and its subtree; - any element carrying an inline
stylethat setsdisplay:none, and its subtree.
No CSS resolution. A cascade engine would not be deterministic across implementations, so a class that hides an element in an external stylesheet does not hide it here. This standard states that plainly rather than implying a fidelity it does not have. Fixture 037-hidden-text-facts pins the rule in the direction that costs a page points.
JSON-LD script contents are not counted text. They are surfaced separately. The alternative reading — that <script type="application/ld+json"> is visible text because it is not "a script" — breaks two mechanisms this standard depends on: D5.3 checks that structured data agrees with visible text, and "corroborated" means observed in ≥2 sources. If JSON-LD were visible text, both would be satisfied by a single JSON-LD block, trivially and always.
The tokenizer MUST NOT throw for any input string — unterminated tags, unterminated comments, stray <, lone surrogates, deeply nested elements, binary noise. A scanner that can be crashed by the page it is scanning is a denial-of-service endpoint.
4.8 Money
Text is converted to a currency and an integer number of minor units (USD 14800 for $148.00), or to null. Ambiguous input MUST return null, and a null price is simply not a fact — it costs coverage points and nothing else. A wrong price would be published as truth and would make a parity check accuse a publisher of a divergence they did not commit. Every rule resolves toward null.
Supported formats, each of which has a test:
| Form | Examples |
|---|---|
| Leading symbol | $1,499.00 · €49 · ¥4,900 |
| Trailing symbol | 1.499,00 € · 49 zł |
| ISO code prefix / suffix | USD 1499 · 1499 USD |
| Comma and period as both group and decimal separator | disambiguated by position and by the minor-unit width of the resolved currency |
| Space / NBSP / narrow NBSP / apostrophe group separators | 1 499,00 · 1'499.00 |
| Zero-decimal currencies | ¥4,900 → 4900, not 490000 |
| Ranges | $148.00 – $198.00 → the lower bound, qualified: true |
| Qualifiers | From $9, $9+ → the stated bound, qualified: true |
qualified: true values are excluded from the D2.4 and D5.3 comparisons. "From $9" in Markdown against "$9 – $40" in HTML is one page saying one thing in two shapes, and flagging it would be a false accusation.
Implementations MUST NOT apply Unicode normalisation (NFC/NFKC) in the money path: normalisation is deterministic only for a fixed Unicode version, and the Unicode version is a property of the host's ICU build — the same class of unversioned input as a third-party parser. Every special-cased character is enumerated in a pinned table.
5. Dimensions and checks
5.1 The allocation
Fixed. 100 points. Identical for every page kind. No conditional weights, no redistribution, and no heuristic gate over measured points. Redistribution is a gaming vector — make the dimension you are bad at inapplicable and everything else inflates. A heuristic that can zero measured points makes those points heuristic.
| Dim | Weight | Check | id | Pts | Basis |
|---|---|---|---|---|---|
| D1 Retrievability | 20 | D1.1 Reachable on the agent path | retrievability.reachable | 8 | measured |
| D1.2 Robots policy for assistants | retrievability.robots-policy | 6 | measured | ||
| D1.3 Core facts present without JavaScript | retrievability.render-independence | 6 | heuristic | ||
| D2 Machine representation | 20 | D2.1 Machine representation on Accept negotiation | machine-representation.negotiated-response | 10 | measured |
| D2.2 Declared alternate representations | machine-representation.declared-alternates | 4 | measured | ||
D2.3 Vary: Accept declared correctly | machine-representation.vary-accept | 3 | measured | ||
| D2.4 Same substance across representations | machine-representation.substance-parity | 3 | heuristic | ||
| D3 Fact coverage | 20 | D3.1 Core facts for this page kind | fact-coverage.core-facts | 14 | heuristic |
| D3.2 Context cost relative to this page kind | fact-coverage.context-cost | 6 | measured | ||
| D4 Fact position | 10 | D4.1 Byte offset of the first core fact | fact-position.first-core-fact-offset | 5 | heuristic |
| D4.2 Core facts inside the front window | fact-position.front-window | 5 | heuristic | ||
| D5 Structured data | 15 | D5.1 Structured data present and parsing | structured-data.present | 5 | measured |
| D5.2 Recognised type with its required properties | structured-data.required-properties | 7 | measured | ||
| D5.3 Structured data agrees with visible text | structured-data.text-agreement | 3 | heuristic | ||
| D6 Contract & discovery | 15 | D6.1 Canonical URL present and self-consistent | contract-discovery.canonical | 4 | measured |
| D6.2 Cache validators | contract-discovery.cache-validators | 3 | measured | ||
| D6.3 llms.txt present and spec-shaped | contract-discovery.llms-txt | 2 | measured | ||
| D6.4 Sitemap declared in robots.txt | contract-discovery.sitemap | 2 | measured | ||
| D6.5 Declared machine endpoint | contract-discovery.machine-endpoint | 4 | measured |
Check ids have the normative shape <dimension>.<check>. A result MUST carry the dimension explicitly as well; consumers MUST NOT parse an id string to group results.
earned is an integer in [0, weight] for every check, in every result, always.
5.2 The measured/heuristic split
Measured 64 · Heuristic 36 · spec ceiling on heuristic weight: 40.
The ceiling is a constraint on this standard, not a description of it: a future MINOR that pushes heuristic weight above 40 is out of conformance with its own spec and fails the property test that computes it.
The 64/36 figure is honest because D2.4 and D5.3 are scored checks, not gates. A parity failure loses its own 3 points and raises a warn; it does not touch D2's other 17 measured points. The same for D5.3 against D5.2. The property that must hold, and which is asserted over the whole corpus, is:
The sum of every check whose earned value can be zeroed or scaled by a heuristic equals the published heuristic weight.
Two gates exist, and both point the safe way: D2.3 and D2.4 are gated on D2.1, which is a measured check gating others. That direction is what makes the published ceiling in §8 provable. The forbidden direction — a heuristic gating a measured check — appears nowhere in this standard.
5.3 D1 — Retrievability (20)
5.3.1 D1.1 Reachable on the agent path — 8, measured
8 = 4 + 2 + 2:
| Condition | Pts |
|---|---|
| Agent-probe status is 2xx | 4 |
Redirect hops ≤ maxRedirects (3) | 2 |
| Both probes resolved to the same final URL | 2 |
The third condition is how "no Accept-conditional redirect" is measured. When there is no browser capture in the bundle it scores 0, with an evidence line saying not comparable — no browser-control capture in this bundle. Awarding it by default would make a measured check heuristic; deducting it silently would be unexplained. The evidence line is the difference, and this rule applies to every sub-condition in this standard that compares the two probes.
5.3.2 D1.2 Robots policy for assistants — 6, measured
6 = 3 + 1 + 1 + 1:
| Condition | Pts |
|---|---|
| The assistant audience is not disallowed | 3 |
robots.txt parses with zero malformed lines | 1 |
Zero rules appear before any User-agent: line | 1 |
A Sitemap: line is declared | 1 |
If the assistant audience is disallowed by a group that does not name an assistant token, the whole check scores 0 and a robots-contradiction warn is raised (§7.3). An absent robots.txt scores 5/6: nothing is disallowed, nothing is malformed, nothing is orphaned, and no sitemap is declared.
The training audience is read, reported, and never scored (§7.4).
5.3.3 D1.3 Core facts present without JavaScript — 6, heuristic
Scored over the structural representation. Let r = ⌊100 × (distinct core fact kinds found in the structural representation) ÷ (core profile size)⌋.
r at least | Pts |
|---|---|
| 100 | 6 |
| 75 | 5 |
| 50 | 3 |
| 25 | 1 |
| — | 0 |
Plus a <noscript> floor of 1: a page whose facts are all client-rendered but which ships a non-empty <noscript> has told a JS-less caller something. earned = max(band, noscript ? 1 : 0). Fixtures 044-spa-empty-root (0/6) and 045-noscript-fallback (1/6) are the pair.
When there is no structural representation at all, D1.3 scores 0 with the evidence line none in this bundle — not evaluated.
Below r = 50 a render-dependent info flag is raised, carrying the ratio.
This check is labelled heuristic in the result and MUST be labelled heuristic in any UI, with its inputs shown. ARS never executes JavaScript, so this measures the served HTML rather than what a browser would render. An optional deep render profile MAY be offered as a diagnostic and MUST NOT change the score.
5.4 D2 — Machine representation (20)
5.4.1 D2.1 Machine representation on Accept negotiation — 10, measured
All or nothing. Earns 10 when both hold on the agent probe:
- The
Content-Typemedia type classifies asmarkdown(text/markdown,text/x-markdown),text(text/plain), orjson(application/json,application/ld+json, or any*/*+json); and - the body does not look like HTML — tested on the first 1 KiB, independently of what the header claims.
A response that declares text/markdown and returns an HTML document earns 0, and so does a response that returns real Markdown labelled text/html (fixture 036-negotiation-wrong-content-type). The body's opinion and the header's claim must agree.
A page whose bytes are HTML is still tokenized whatever the header said, so a mislabelled content type costs 10 points on D2.1 and does not additionally cost fact coverage. One mistake is charged once.
5.4.2 D2.2 Declared alternate representations — 4, measured
All or nothing. Earns 4 when at least one typed alternate is declared, from either probe:
- a
Linkheader withrel="alternate"and atypeparameter; or - a
<link rel="alternate">in the structural document with bothtypeandhref.
An alternate with no type does not count. Untyped alternates are language variants (hreflang), which every international site ships, and paying 4 points for them would make the check meaningless.
Link headers are parsed per RFC 8288. The separator is a comma and target IRIs contain commas, so <https://x/a,b>; rel="alternate" is one link; an implementation that splits on , silently drops alternates that were declared.
5.4.3 D2.3 Vary: Accept declared correctly — 3, measured
All or nothing, and 0 unless D2.1 earned points. Earns 3 when either probe's response carries Accept in its Vary header token list.
The gate is the point: Vary: Accept on a response that does not vary by Accept is a claim about caching that is not true, and fixture 035-vary-without-negotiation pins it at 0. When D2.1 earned points and Vary: Accept is absent, a vary-missing warn is raised — a shared cache can serve the Markdown to a browser or the HTML to an agent.
5.4.4 D2.4 Same substance across representations — 3, heuristic
All or nothing, and 0 unless D2.1 earned points and a browser capture exists.
Compared kinds are exactly four: price, currency, availability, title. Never description, never free text. A prose difference between a Markdown summary and an HTML page is a rendering difference, and calling it a substance divergence would be an accusation the evidence does not support.
The comparison rule is asymmetric, deliberately:
The value one side would publish — its lowest-offset unqualified observation, the same choice the published fact set makes — MUST appear somewhere among the values the other side stated.
Both symmetric rules are wrong, and each is wrong in a way that matters. Requiring the two representatives to be equal produces false accusations on ordinary pages: a real product page that names a backordered variant above the line saying the product is in stock states two availabilities, and both statements are true. Accepting any shared value lets a real divergence hide behind an incidental match, because "Free shipping over $50" appears in both representations. The asymmetric rule catches the second and not the first, which is the conservative direction: a missed divergence costs a publisher nothing, and a false one is published next to their name.
Scoring:
- No comparable kind present in both → 0, with
Scored: no — a comparison needs at least one comparable fact in both representations. This is a real cost for stating nothing in common, and it is not a flag. - Values agree → 3.
- Values disagree, and no confirming probe is present → 0, and no flag. One observation cannot distinguish a cloaker from an inventory change.
- Values disagree, and a confirming probe shows the difference did not reproduce → 3, recorded as a value that changed between requests.
- Values disagree and the difference reproduced on the confirming probe → 0 and a
substance-divergencewarn.
The flag message is an observation, never an accusation, and MUST remain one: "The agent representation reported price USD 9900; the HTML representation reported USD 14800 at capture time, on two captures taken apart." No assertion about which value is correct, and none about intent.
Known limitation: the confirming probe is required to be taken ≥30 s after the first, but the evidence bundle carries no per-probe timestamp, so the delay is a contract on the probe rather than something score() can verify. A bundle that puts a same-millisecond re-fetch in parityConfirm scores identically. Recorded in Appendix D.
5.5 D3 — Fact coverage (20)
Scored over the agent representation.
- D3.1 Core facts — 14, heuristic. Coverage against the page-kind profile, scaled by a density band. Arithmetic in §6.3.
- D3.2 Context cost — 6, measured. Decoded bytes against the page-kind byte reference. Arithmetic in §6.4.
5.6 D4 — Fact position (10)
Both checks are heuristic, and both are scored over the agent representation. Arithmetic in §6.5.
- D4.1 First core fact offset — 5.
- D4.2 Front window — 5. The denominator is the core facts that are present, not the profile size. Absence is D3.1's job, and charging the same absence twice would make the 100 points non-orthogonal. The evidence prints both numbers so a reader can see which question was asked.
5.7 D5 — Structured data (15)
Scored over the structural representation.
5.7.1 D5.1 Structured data present and parsing — 5, measured
jsonLd = 5 when ≥1 JSON-LD block is present and every block parsed
= 3 when ≥1 block parsed and ≥1 did not
= 0 when blocks are present and none parsed
= — when no blocks are present
other = 5 when ≥1 microdata `itemtype` or RDFa `typeof` scope is present
= — otherwise
earned = max(jsonLd, other, 0)Microdata therefore rescues D5.1 on a page whose only JSON-LD block is broken — but it does not rescue D5.2, which walks JSON-LD only (§5.7.2).
A block that does not parse is not structured data. Fixture 033-invalid-jsonld pins the whole of D5 at 0 for a page whose only block is malformed — and pins that the scorer does not throw.
5.7.2 D5.2 Recognised type with its required properties — 7, measured
7 = 2 + 5:
- 2 for a recognised schema.org type being present at all.
- 5 × (satisfied ÷ required), half-up (§6.6), for required-property completeness.
The scored node is the JSON-LD node whose @type decided the page kind; if none, the first node with any recognised type. Scoring the first node in the document would mark a page incomplete because its BreadcrumbList has no offers.price.
Required properties come from ruleset.requiredProperties (Appendix B), a hand-curated, versioned table shipped inside the package. Its provenance is stated because D5.2 is 7 points and publishers will ask:
- It is not "schema.org validity". schema.org defines no required properties at all.
- It is not a search engine's rich-results guidelines by reference. Those change without a version, and would make ARS scores move without an ARS release.
- It is the core fact set of each page kind expressed in schema.org vocabulary, plus the properties without which those facts are not actionable — a price with no
priceCurrencycannot be used by a buying agent, soproductandservicerequire both.
Notation: a dot is a property path from the scored node, and traversal enters array members, so offers.price matches offers[0].price. Alias handling (openingHours vs openingHoursSpecification, Article.headline vs name) belongs to the extractor and not to this table.
Known limitation: in 0.1, D5.2 walks JSON-LD only. Microdata and RDFa satisfy D5.1 but are not traversed for required properties, so a microdata-only page scores at most 5/15 on D5. Recorded in Appendix D.
5.7.3 D5.3 Structured data agrees with visible text — 3, heuristic
All or nothing. Applies the same asymmetric comparison as D2.4 (§5.4.4) over the same four kinds, but within one representation: the values the JSON-LD would publish against the values the visible page stated.
- Nothing comparable → 0 (no flag).
- Agreement → 3.
- Disagreement → 0 and a
structured-data-divergencewarn.
Fixture 034-jsonld-price-mismatch is the canonical case: JSON-LD says $99, the page says $148.
5.8 D6 — Contract & discovery (15)
5.8.1 D6.1 Canonical — 4, measured
4 = 2 + 1 + 1: present (2), absolute http(s):// (1), identical across both probes (1). The HTML-side canonical is taken from the browser capture's Link header or <link rel="canonical">, falling back to the agent side; the agent-side canonical is taken from the agent capture only. A mismatch between the two raises canonical-mismatch (warn).
A representation with no canonical cannot be attributed to a page, which is why a Markdown response that declares none loses the third point even when the HTML is perfect (§11.1).
5.8.2 D6.2 Cache validators — 3, measured
3 = 2 + 1, read from the agent capture:
- 2 for a sane
Cache-Control: the header is present, does not containno-store, and either declaresmax-age/s-maxagegreater than zero, or declaresno-cache/must-revalidateand a validator exists. Revalidation only means something with a validator. - 1 for an
ETagor aLast-Modified.
5.8.3 D6.3 llms.txt — 2, measured
1 for a file that exists and is non-empty at /llms.txt; 2 when it is spec-shaped — an H1 heading and at least one Markdown link. A file with a title and no links tells an agent nothing.
This check is worth 2 of 100, deliberately. The reasoning, stated as the editors' opinion and offered as an opinion rather than as a finding:
We weight
llms.txtlow. Content negotiation is the mechanism we have seen change what a caller actually receives; we have observed no citation lift fromllms.txtalone.
No study is cited for that, because we have none to cite. Rebilder's marketing material carries a byte-reduction figure that does not appear anywhere in this standard, and deliberately so: asking third parties to implement a standard on the strength of an uncited number is a different act from putting that number in an advertisement. If you disagree with the weight, it is 2 points of 100 and the ruleset is published, so the disagreement is cheap to quantify.
5.8.4 D6.4 Sitemap — 2, measured
All or nothing: a Sitemap: line in robots.txt.
5.8.5 D6.5 Declared machine endpoint — 4, measured
All or nothing over any one declared endpoint. Declarations are read from:
- a
Linkheader, or a<link>in the structural document, whoserelis one ofservice-desc,service-doc,api,describedby,ucp,mcp; - a
Linkheader or<link>whosetypeis one ofapplication/rss+xml,application/atom+xml,application/feed+json,application/json,application/ld+json,application/vnd.api+json,application/schema+json,application/graphql,application/openapi+json,application/yaml,text/calendar; - a Markdown link in
/llms.txtwhose URL contains/api,/mcp,/graphql,/openapi,/.well-known/,.json,.ics,.rss, or.atom; - a non-empty
/.well-known/ucp.
Nothing is guessed (§3.7).
6. Scoring arithmetic
6.1 Integer arithmetic only
No floating point appears anywhere in the scoring path. Cross-language conformance is impossible with float rounding drift.
- All inputs are non-negative, so exact integer division is
⌊a ÷ b⌋ = (a − (a mod b)) ÷ b. Implementations SHOULD use this form rather thanfloor(a / b)on a double quotient, whose intermediate value can land on the wrong side of an integer boundary. - Half-up rounding is
(a + ⌊b ÷ 2⌋) ÷ bin integer division, writtenround½(a, b). - Division by zero yields 0.
round(),× 0.25and× 0.5are forbidden in the scoring path.
Two table encodings are used, and each is declared per table:
- AT-LEAST — descending
[threshold, value]pairs; the first pair withthreshold ≤ inputwins; no match scores 0. - AT-MOST — ascending
[threshold, value]pairs; the first pair withinput ≤ thresholdwins; no match scores 0.
A malformed or absent table scores 0. It MUST NOT invent points.
6.2 Fact weighting, in quarter-units
coreUnits = 4 × (corroborated core facts) + 2 × (uncorroborated core facts) extendedUnits = 1 × (corroborated extended facts) + 0 × (uncorroborated extended facts) denominator = 4 × coreSize coveragePct = min(100, round½(100 × (coreUnits + extendedUnits), denominator))
Full credit is every core fact corroborated. An uncorroborated extended fact contributes nothing — this is the single definition of "extended = ¼ credit", and it is the only one in this standard.
coveragePct is clamped at 100: a page with every core fact corroborated and several extended facts exceeds the denominator and gains nothing further. That clamp is a deliberate cap on stuffing (§9).
6.3 D3.1 — core-fact coverage (14 pts)
factsPerKiB100 = ⌊(coreFound × 100 + extendedFound × 25) × 1024 ÷ max(1, bytes)⌋
bytes is the decoded byte length of the agent representation. The max(1, bytes) clamp is normative and replaces the max(1, bytes ÷ 1024) form, which let a 200-byte stub top the density band.
Density band, AT-LEAST over factsPerKiB100 (hundredths of a fact per KiB):
| at least | densityPct |
|---|---|
| 200 | 100 |
| 100 | 85 |
| 50 | 70 |
| 20 | 50 |
| 5 | 30 |
| 0 | 15 |
earned = ⌊(14 × coveragePct × densityPct + 5000) ÷ 10000⌋
Density saturates at ≥ 2 facts per KiB. Above that threshold, adding bytes cannot buy a band the page could not already reach, and adding fabricated facts cannot buy anything at all because coverage is capped at the profile.
6.4 D3.2 — context cost (6 pts)
ratio = ⌊100 × bytes ÷ byteReference[pageKind]⌋
AT-MOST over ratio:
| at most | Pts |
|---|---|
| 100 | 6 |
| 200 | 5 |
| 500 | 3 |
| 1500 | 1 |
| — | 0 |
6.5 D4 — position (5 + 5)
D4.1, AT-MOST over the byte offset of the first core fact:
| at most | Pts |
|---|---|
| 512 | 5 |
| 2048 | 4 |
| 8192 | 2 |
| 32768 | 1 |
| — | 0 |
No core facts found → 0.
D4.2. The front window is one expression, in integer division:
window = max(2048, ⌊bytes ÷ 5⌋)
inside = |{ core facts with offset < window }|
pct = ⌊100 × inside ÷ (core facts found)⌋The 0.2 × bytes form is deleted from this standard: two implementations rounding a float disagree at the boundary.
AT-LEAST over pct:
| at least | Pts |
|---|---|
| 90 | 5 |
| 70 | 4 |
| 50 | 3 |
| 25 | 1 |
| — | 0 |
No core facts found → 0.
6.6 D5.2 — completeness
completeness = round½(5 × satisfied, required) // 0 when the table row is empty → 5 earned = 2 + completeness // when a recognised type is present earned = 0 // when none is
6.7 Total and bands
The score is the sum of the six dimension totals, each of which is the sum of its checks' earned. There is no normalisation step, no curve, and no rounding at the end, because nothing fractional ever entered.
| Grade | Range | Label | Meaning (descriptive only) |
|---|---|---|---|
| A | 90–100 | Agent-native | Facts are in a machine representation, complete for this page kind, front-loaded, and cheap to fetch. |
| B | 75–89 | Agent-friendly | All core facts are present and findable; the agent pays more bytes than it needs, or one signal is missing. |
| C | 60–74 | Readable with effort | Core facts are extractable from HTML, at high cost. |
| D | 40–59 | Partial | Some core facts for this page kind were not found in the fetched response. |
| F | 0–39 | Not extractable | Few or no core facts for this page kind were found in the fetched response. |
The band labels and meanings are descriptive only and their phrasing is load-bearing. Three formulations that appeared in earlier drafts are deliberately gone and MUST NOT be reintroduced:
- "expect wrong or missing details in AI answers about this page" — a predictive assertion about third-party model output that ARS does not measure;
- "effectively closed to agents" — falsifiable, and wrong for a site at 39 serving clean HTML;
- "…and can trust it" — asserts veracity, and ARS measures format, not truth.
6.8 Non-grades
Three outcomes carry score: null, grade: null, bandLabel: null, and an empty recommendation list. None of them is an F.
| Outcome | Reason |
|---|---|
opt-out | A deliberate, well-formed robots disallow for the assistant audience. A choice, not a failure. Never ranked, never listed |
unscored | blocked-at-edge — 403/401/429 or a challenge at the edge. The evidence is itself the finding |
unscored | unreachable, non-2xx, too-many-redirects, robots-disallow-scanner, robots-unavailable, truncated-evidence, evidence-incomplete |
Gate order is normative, because two gates can both be true:
robots-disallow-scanner— a statement about our conduct. If we were not permitted to fetch, nothing else in the bundle should be reported as a finding. Onlyvantage: 'public'is gated (§7.5).robots-unavailable— a policy we could not read is not a policy we may assume.opt-out— the site's own decision, reported before any failure of ours, because it is a choice and not a failure.- Probe failure → blocked status → non-2xx → redirect limit → truncated body → missing body.
A non-grade result still carries all six dimensions with their weights and every check at earned: 0 with an evidence line naming the reason. It MUST NOT carry sub-scores: publishing D5 for a site that opted out would be scoring it by another name, and a ranking could be reconstructed from the parts.
6.9 Cost report
Every scored result carries: HTML bytes, negotiated bytes (or null), approximate tokens for each, a reduction figure, the first core-fact offset, and truncated.
- Decoded UTF-8 bytes are the normative measurement. The token figures are
⌊bytes ÷ 4⌋, which is not a token count, and this standard says so: they are heuristic and MUST be rendered with≈and an "est." label wherever they appear. - The reduction figure is an integer percentage of HTML bytes eliminated by the machine representation (
94means "94% fewer bytes"). A multiple (N.N×) would be a float, and this is the same information with nothing for two implementations to disagree about. No byte-reduction multiple appears anywhere in this standard: Rebilder's marketing material carries one, it is not cited here, and a standard is not the place to argue a number a reader cannot check.
6.10 Flags
Flags are observations attached to a result. Severity is info or warn. There is no critical severity, and there MUST NOT be one: no check in this standard publicly accuses anyone. Flag order in the output array is pinned so that two runs over the same evidence emit the same array.
substance-divergence, structured-data-divergence, robots-contradiction, paywalled, render-dependent, scanner-blocked, training-opt-out, assistant-opt-out, vary-missing, canonical-mismatch, body-truncated, vantage-variance.
7. Crawler policy
7.1 The three audiences
| Audience | Treatment |
|---|---|
assistant | The audience ARS scores. User-triggered fetches on behalf of a person asking a question |
training | Strictly neutral. Never lowers the score, under any circumstance |
search | Reported only. An input to robots-contradiction, never to a point |
7.2 Pinned tokens
Matching is case-insensitive (RFC 9309 §2.2.1); the casing below is the published casing.
- assistant —
ChatGPT-User,Claude-User,Claude-SearchBot,DuckAssistBot,Meta-ExternalFetcher,MistralAI-User,OAI-SearchBot,Perplexity-User - training —
AI2Bot,Applebot-Extended,Bytespider,CCBot,ClaudeBot,Diffbot,FacebookBot,Google-Extended,GPTBot,Meta-ExternalAgent,PanguBot,Timpibot,anthropic-ai,cohere-ai,omgili - search —
Applebot,Baiduspider,Bingbot,DuckDuckBot,Googlebot,Slurp,YandexBot
These lists are versioned data, not a heuristic. Adding a token changes the score of every site that named it, so additions land as a MINOR.
7.3 robots state → outcome
robots state for the assistant audience | Outcome |
|---|---|
| Allows, or is unspecified | Scored |
| Deliberate + well-formed disallow — a group that names an assistant token, zero malformed lines, zero orphan rules | opt-out, score: null, no letter grade, never listed |
Accidental or ambiguous — a blanket User-agent: * disallow, conflicting groups, malformed syntax | Scored, D1.2 = 0, robots-contradiction warn, remedy worth 6 |
Persistent 5xx on /robots.txt | unscored / robots-unavailable |
Disallows rebilder-ars (and vantage is public) | unscored / robots-disallow-scanner — we obey it and we say so |
Consent has to be specific. A blanket User-agent: * / Disallow: / on a site with full commerce markup is the accidental case, not an opt-out: it is scored with the contradiction flagged, and the flag message says what it observed rather than what it suspects. Conversely, a site cannot be measured after opting out on purpose. Fixtures 020 (deliberate) and 023 (blanket) are the pair.
7.4 Training-crawler neutrality — and its proof
Blocking model-training crawlers while allowing assistant fetches scores identically to a fully open site.
This is not a policy statement; it is a property of the corpus, and it is the first thing a hostile reviewer should test.
Fixtures 021-robots-training-optout-only and 022-robots-open are the same page, the same Markdown and HTML bodies, the same headers, the same vantage, the same llms.txt and /.well-known/ucp misses. 021's robots.txt adds one group naming GPTBot, CCBot, Google-Extended and anthropic-ai with Disallow: /; 022's does not. Both score 98, grade A.
Diffing the two expected.json files touches exactly three things, none of which is a point: policy.audiences.training, policy.trainingOptOut, and one info-severity training-opt-out flag whose stated effect on the score is "none". Every dimension, every check, every earned point, the cost figures and the recommendation list are identical.
Because two files that happen to agree today prove only that they agree today, the property is also asserted directly: the reference implementation re-reads both bundles, checks that they differ only in robots.txt — otherwise an identical score would prove nothing about robots.txt — and then compares the scored shape field by field, including the flag's severity. A warn there would be the score's disapproval wearing a different hat, which is precisely what this section forbids.
7.5 The scanner token and vantage
We publish a scanner token, rebilder-ars, so that a site can refuse us specifically without refusing assistants. Only a group that literally names rebilder-ars triggers the refusal; a blanket User-agent: * never does.
vantage: 'self' bypasses the scanner-token check. The owner can consent for their own origin: a publisher running a local CLI against their own store, or previewing in their own console, is not a third party who needs robots.txt's permission to look at their own site. Without this carve-out, the most careful publishers — the ones who block unknown crawler tokens — would be told their own site cannot be scored, which is a product regression dressed as rigour.
The carve-out is not a route around anyone else's wishes:
vantageis set by the impure half from proof of who is asking, it is covered byevidenceHash, and it is printed on every result.- It changes nothing about publication. Nothing derived from an authenticated, merchant-sourced, or locally-run scan is ever published, whatever its score.
Fixtures 025 and 026 are byte-identical bundles differing only in vantage: unscored / robots-disallow-scanner and A (98) respectively.
7.6 Constraints on publication
These bind any implementation that publishes ARS results about third parties. They are part of this standard because a measurement standard that is silent about publication is an invitation.
- Nothing about a named third party may be published without verified domain-owner opt-in. Aggregate distributions that name nobody are unconstrained.
- Removal MUST be free, self-serve, permanent, and never purchasable. No outbound sales contact may be made on the basis of a score.
- Any aggregate published about a group MUST meet a floor of 25 independent subjects per bucket. This floor is not a tunable.
- A badge or embed MUST always name its subject, MUST be served from stored data, MUST be architecturally incapable of triggering a probe, and MUST degrade to "unverified" after a fixed staleness window — otherwise a site that starts blocking the scanner freezes at its best score forever.
- A public result page MAY publish structure and numbers — tag histograms, byte offsets, content types, sizes, check evidence values — and MUST NOT publish prose excerpts of the scanned page.
8. The designed ceilings
These are consequences of the published weights, stated here as intentional rather than left for someone to discover, and disclosed next to the conflict-of-interest note in §0.1.
| Missing | Points lost | Ceiling | Band |
|---|---|---|---|
| No content negotiation | D2.1 (10) + D2.3 (3, gated on D2.1) + D2.4 (3, nothing comparable) | 84 | capped at B |
| No valid structured data | all of D5 (15) | 85 | capped at B |
Reaching an A requires both a machine representation and valid structured data.
Two things about this are worth reading carefully rather than accepting or dismissing:
- It is provable from the published ruleset alone. Nothing in this claim depends on trusting the implementation: take the weights table in §5.1, remove the rows, add up the rest.
- Rebilder sells software that satisfies both. That is the conflict, it is disclosed at the top of this document, and it is the reason the ruleset, every threshold, and the whole corpus are published rather than described.
The counter-argument a reviewer should make is that a standard whose top band requires the author's product category is self-serving. The editors' answer is not that the conflict does not exist; it is that the two requirements are the two mechanisms by which a caller gets a fact out of a page without guessing, that both are open standards implementable by anyone with no Rebilder software involved — Content-Type negotiation is HTTP, and JSON-LD is schema.org — and that the disagreement is quantifiable because the numbers are printed.
9. Adversarial mitigations
Written honestly, including the ones that cannot be mitigated.
| Attack | Mitigation |
|---|---|
Serve clean Markdown to Accept: text/markdown and different substance to humans | D2.4 substance parity. Facts from both probes, normalised, compared on exactly four kinds — price, currency, availability, title — never description or free text. A reproduced mismatch scores 0 on D2.4 and raises a warn. Divergence MUST reproduce on a third confirming probe ≥30 s later before the flag is set: inventory and price genuinely change between two sequential requests |
Cloak against rebilder-ars specifically | Not detectable. Disclosed (§1.2, §3.2). Both probes identify honestly, so a site serving identical fabricated substance to both scores as if truthful. "ARS is not a cloaking-compliance certification" is printed next to D2.4 and on every badge |
| Keyword-stuff with fabricated facts | Coverage is capped at the profile — only facts the resolved profile names are extracted at all — and coveragePct is clamped at 100; density saturates at ≥ 2 facts/KiB. Fixtures 031 and 032 are the same kettle 691 bytes apart and produce identical scores across all nineteen checks |
| JSON-LD spam | D5.2 scores required-property completeness for the detected kind against a pinned table; D5.3 independently checks agreement with visible text. Fixture 034: JSON-LD $99 against a page saying $148 → D5.3 = 0 and a structured-data-divergence warn |
llms.txt as a checkbox | Worth 2 of 100, with the reasoning printed as an opinion (§5.8.3) |
| False precision — "you gave me 81 and my competitor 83" | Bands headline; no decimals ever; every check reports its basis and its raw evidence; heuristic weight is capped at 40 and published. ARS is ordinal (§2.6). A property test pins every band boundary at 39/40, 59/60, 74/75, 89/90 |
| Penalising a deliberate opt-out | opt-out is a non-grade, not an F. Training-only opt-outs are strictly neutral (§7.4) |
| Paywalled content | A declared paywall (isAccessibleForFree: false, HTTP 402, WWW-Authenticate) raises an info flag and is not a penalty. Undeclared soft paywalls are undetectable without impersonation, and we will not impersonate. In 0.1 the declared-paywall exemption of the gated part from coverage and position is not implemented — Appendix D |
| Bot management blocks the probe | unscored / blocked-at-edge, never a low grade. The evidence is itself the finding. A vantage: 'self' run produces an authoritative score from inside the perimeter |
| Vantage manipulation — serve well only to known scanner egress | Not mitigated in 0.1. A vantage-variance flag exists in the vocabulary; enforcement (re-probing from a second egress) is deferred. Because 0.1 publishes nothing about non-consenting third parties (§7.6), the exposure this creates is bounded |
| Threshold gaming | Exact thresholds are published and every sub-score is reported. There is nothing to reverse-engineer |
| Page-kind shopping | unknown still requires a recognised type for D5.2 and has no coverage advantage; concealing the kind removes the concealed facts from the scored set (§4.4) |
| "Render dependence is a guess" | It is, and it says so. D1.3 is labelled heuristic in the result and in the UI, with its inputs shown; <noscript> earns partial credit; an optional deep render profile is a diagnostic and does not change the score |
| "chars ÷ 4 is not a token count" | Correct. Decoded UTF-8 bytes are the normative measurement; approximate tokens carry ≈ and an "est." label everywhere (§6.9) |
Three that this standard does not mitigate, stated plainly: User-Agent-targeted cloaking, undeclared soft paywalls, and egress-targeted vantage manipulation. Each is undetectable without either impersonating another party's crawler or fetching from infrastructure we do not disclose, and this standard chooses to do neither and to say so.
10. Conformance
10.1 Corpus layout
conformance/NNN-name/
evidence.json the INPUT — an ArsEvidence bundle, response bodies included
expected.json SPEC-DERIVABLE fields only. A second implementation MUST reproduce this
identity.json rulesetHash / corpusHash / evidenceHash — properties of the reference artifact.
A second implementation is explicitly NOT required to match them
README.md one paragraph on what the fixture provesEvery directory is a fixture. A runner MUST discover them by globbing and MUST treat a directory with a missing or unparseable evidence.json, a missing expected.json, or a missing README.md as an error, never as a skip. A corpus that quietly ignores half of itself reports green while proving nothing.
10.2 Why the expected.json / identity.json split exists
This split is what makes independent certification possible at all, and it is the most important structural decision in this section.
expected.json contains only what the spec determines. Every number and every identifier: earned per check and per dimension, the grade, the score, the band label, the page kind and its confidence, the fact set with offsets and corroboration, the flags raised with their severity and basis, the policy decisions, the cost report, and the recommendation arithmetic. A conformant implementation in any language MUST reproduce this file exactly.
It deliberately excludes:
- The three hashes. They live in
identity.json. specVersion, which is pinned by the ruleset rather than derived.- Every human-facing string — check labels, evidence lines, flag messages, recommendation titles and copy. Those are the editors' to reword in a PATCH, and a corpus that pinned them would turn a copy edit into a spec change.
identity.json contains only what is a property of our artifact: rulesetHash, corpusHash, evidenceHash. A second implementation is not required to match these. If certification required matching our hashes, it would require our exact serialisation of our exact object graph, and "conformant" would mean "a transliteration of the reference implementation". It does not. It means: given this evidence, produce these numbers.
Note that rulesetHash and evidenceHash are asserted by the reference implementation's own runner, because they are properties of inputs that do not move. corpusHash is recorded and not asserted per fixture: it is a digest over the whole corpus, so pinning it per fixture while the corpus is open would make every new fixture break every old one.
10.3 The fixture list
Golden pair — the same 91 226-byte HTML body, one variable changed:
| Fixture | Result |
|---|---|
001-pdp-gateway-md | A (90) — a product page with content negotiation installed |
002-pdp-raw-html | D (54) — the same page, raw |
Universality — every page kind, and the byte-reference proof:
010-place-hours-structured (C 70) · 011-place-hours-image-only (C 66) · 012-article-paywalled (B 77) · 013-service-pricing (C 72) · 014-document-openapi (B 79) · 015-unknown-untyped (C 65) · 016-collection (C 72) · 017-faq (C 61) · 018-article-longform (B 77)
Policy — read first by any hostile reviewer:
020-robots-deliberate-optout (opt-out, score: null) · 021-robots-training-optout-only and 022-robots-open → identical A (98) · 023-robots-accidental-blanket (A 92, D1.2 = 0) · 024-robots-persistent-500 (unscored) · 025-robots-blocks-our-scanner (unscored) · 026-self-vantage-bypasses-scanner-token (A 98)
Adversarial:
030-accept-conditional-divergence (A 91 + substance-divergence) · 031-keyword-stuffed and 032-density-baseline → identical C (74) · 033-invalid-jsonld (D 52) · 034-jsonld-price-mismatch (C 70) · 035-vary-without-negotiation (C 71) · 036-negotiation-wrong-content-type (B 75) · 037-hidden-text-facts (D 49) · 038-gateway-install-not-flagged-as-cloaking (A 100, zero flags)
Error and edge:
040-blocked-403 · 041-timeout · 042-redirect-chain-3 (C 71) · 043-body-over-cap-truncated · 044-spa-empty-root (F 29) · 045-noscript-fallback (F 30) · 046-paywall-declared (C 66) · 047-non-html-content-type (F 25) · 048-shift-jis-charset (C 64)
Properties:
050-byte-identical-replay (C 74, with volatile headers present) · 051-multi-value-headers (A 98) — both are corpus directories. The remaining named properties — weights summing to 100 with earned ≤ weight; heuristic-controlled weight ≤ 40; band boundaries at 39/40, 59/60, 74/75, 89/90; no network; no clock; tokenizer fuzz; profile totality and fact-name typing; the money-parser format list — are properties of the ruleset, the profiles, the parsers or the build rather than of any evidence bundle, and are asserted as unit tests in the reference implementation rather than as fixture directories.
10.4 Running the corpus
In the reference implementation the corpus runs as part of the package test suite. Regeneration is available and is deliberately hostile to misuse: it rewrites expected.json and identity.json from the current implementation and then fails the run, so an update can never be mistaken for a pass. Read the diff before committing it — blessing output you have not checked is how a corpus stops encoding the spec and starts encoding the bug.
10.5 Claiming conformance
An implementation MAY describe itself as ARS 0.1 conformant when all of the following hold:
- For every fixture in the corpus, it produces
expected.jsonexactly, givenevidence.jsonand the ARS 0.1.0 ruleset. - It performs no floating-point arithmetic in the scoring path.
- Its scoring function performs no network access, reads no clock, consumes no randomness, and invokes no language model.
- Every result it emits carries
specVersion,rulesetHash,corpusHashandevidenceHash, and every check carries itsbasis. - Every surface it renders that displays a grade carries the §0.2 measurement notice verbatim.
- If it publishes results about third parties, it satisfies §7.6.
An implementation that meets 1–4 but not 5 is a conformant scorer and a non-conformant publisher, and MUST NOT describe its output as an ARS grade in public.
11. Worked examples
Each of these derives a published fixture end to end from the rules in this document. They exist so an outside team can verify that the corpus encodes this standard rather than merely encoding our code. If any arithmetic below disagrees with the corresponding expected.json, the corpus wins and this document has a defect.
All five derivations use only: the profile table (§4.2), the weights table (§5.1), the sub-point splits (§5.3–§5.8), the arithmetic in §6, and the fields published in the fixture's own expected.json and evidence.json.
11.1 001-pdp-gateway-md → A (90)
A product page served through a content-negotiation gateway. The agent probe receives 1 490 bytes of Markdown (text/markdown; charset=utf-8, Vary: Accept, Cache-Control: public, max-age=300, s-maxage=600, an ETag, no Link header); the browser probe receives the same page's 91 226 bytes of HTML. robots.txt carries a User-agent: * group disallowing eight administrative paths — none of which matches this URL — a Crawl-delay: 1 line, a second group for an ad crawler, and a Sitemap:. /llms.txt returns 200; /.well-known/ucp returns 404.
Page kind is product at high confidence, from the HTML's JSON-LD @type. Profile: core = title, price, currency, availability; extended = brand, sku, shipping, returns, description; byte reference 8192.
Published facts (all from the negotiated source, all corroborated by the HTML's JSON-LD): title @0, brand @101, currency @133, price @133, availability @161, shipping @186, returns @323, sku @1087. Description is absent from the Markdown.
D1 — 20/20. D1.1: 200 (4) + 0 hops ≤ 3 (2) + both probes ended at the same final URL (2) = 8. D1.2: assistant not disallowed (3) + 0 malformed lines (1) + 0 orphan rules (1) + Sitemap: declared (1) = 6. Crawl-delay is ignored by the parser and does not count as a malformed line (§3.6). D1.3: the structural representation is the HTML, which states all 4 core facts → ⌊100 × 4 ÷ 4⌋ = 100 → band 100 → 6.
D2 — 16/20. D2.1: text/markdown is a machine media type, the body is not HTML → 10. D2.2: no Link header on either response and no typed <link rel="alternate"> in the HTML → 0. This is a gap in the gateway, not in the page. D2.3: Vary: Accept present, and D2.1 > 0 → 3. D2.4: comparable kinds present in both (price, currency, availability, title), values agree → 3.
D3 — 20/20. Core: 4 found, all corroborated → coreUnits = 4 × 4 = 16. Extended: 4 found (brand, sku, shipping, returns), all corroborated → extendedUnits = 4. denominator = 4 × 4 = 16. coveragePct = min(100, round½(100 × 20, 16)) = min(100, ⌊(2000 + 8) ÷ 16⌋) = min(100, 125) = **100**. factsPerKiB100 = ⌊(4 × 100 + 4 × 25) × 1024 ÷ 1490⌋ = ⌊512000 ÷ 1490⌋ = 343 → ≥ 200 → densityPct = 100. D3.1 = ⌊(14 × 100 × 100 + 5000) ÷ 10000⌋ = ⌊145000 ÷ 10000⌋ = **14**. D3.2: ratio = ⌊100 × 1490 ÷ 8192⌋ = 18 → ≤ 100 → 6.
D4 — 10/10. D4.1: first core-fact offset 0 → ≤ 512 → 5. D4.2: window = max(2048, ⌊1490 ÷ 5⌋ = 298) = 2048. Core offsets 0, 133, 133, 161 — all inside → pct = ⌊100 × 4 ÷ 4⌋ = 100 → ≥ 90 → 5.
D5 — 14/15. D5.1: every JSON-LD block in the HTML parsed → 5. D5.2: recognised type (2) + completeness. requiredProperties.product = name, offers.price, offers.priceCurrency, offers.availability; the page's offer is an AggregateOffer carrying lowPrice, so offers.price is absent → 3 of 4 satisfied → round½(5 × 3, 4) = ⌊(15 + 2) ÷ 4⌋ = 4 → 2 + 4 = **6**. D5.3: JSON-LD and visible text agree on the compared kinds → 3.
D6 — 10/15. D6.1: canonical present in the HTML (2) + absolute (1) + the Markdown response declares none, so not consistent across probes (0) = 3. D6.2: public, max-age=300, s-maxage=600 is sane (2) + ETag (1) = 3. D6.3: /llms.txt is present with an H1 and Markdown links → 2. D6.4: Sitemap: declared → 2. D6.5: no endpoint declaration anywhere → 0.
Total: 20 + 16 + 20 + 10 + 14 + 10 = 90 → A, "Agent-native". No flags.
Five of the ten missing points are gaps in the middleware rather than in the page — D2.2's 4 and D6.1's 1 both require the Markdown response to declare a Link header it does not currently send. When that is fixed, this fixture's expected.json moves, and that is a MINOR (§2.1).
11.2 002-pdp-raw-html → D (54) — the same page, one variable changed
The controlled experiment. Identical HTML body (91 226 bytes), identical origin, identical robots.txt. The only differences are that the agent probe now receives the HTML too, and the response envelope is the raw theme's: Cache-Control: max-age=0, private, must-revalidate, no ETag, Vary: Accept-Encoding, and /llms.txt returns 404.
Facts published (agent representation = the HTML): title @632 (corroborated), description @726 (uncorroborated), price @1683 (corr.), currency @1736 (corr.), availability @2102 (corr.), brand @2102, sku @2102, returns @31935, shipping @31935 (last four uncorroborated).
D1 — 20/20, unchanged: same status, same hops, same final URLs, same robots.txt, same HTML for D1.3.
D2 — 0/20. D2.1 = 0: the agent asked for Markdown and received HTML. D2.2 = 0: no typed alternate. D2.3 = 0: gated on D2.1 — and note Vary: Accept-Encoding would not have earned it in any case. D2.4 = 0: gated on D2.1, and with one representation there is nothing to compare. This is the 84-point ceiling of §8 in one column.
D3 — 5/20. Core: 4 found, all corroborated → 16 units. Extended: 5 found, none corroborated → 0 units. denominator = 16. coveragePct = min(100, round½(1600, 16)) = 100. Coverage is perfect; the page states everything. factsPerKiB100 = ⌊(400 + 125) × 1024 ÷ 91226⌋ = ⌊537600 ÷ 91226⌋ = 5 → the ≥ 5 band → densityPct = 30. D3.1 = ⌊(14 × 100 × 30 + 5000) ÷ 10000⌋ = ⌊47000 ÷ 10000⌋ = **4**. D3.2: ratio = ⌊9122600 ÷ 8192⌋ = 1113 → ≤ 1500 → 1.
D4 — 9/10. D4.1: 632 → ≤ 2048 → 4. D4.2: window = max(2048, ⌊91226 ÷ 5⌋ = 18245) = 18245; all four core offsets are inside → 100% → 5.
That D4 scores 9 while D3 scores 5 is the orthogonality this standard requires: the facts that exist are near the front, and position is not asked to re-charge an absence that coverage already charged.
D5 — 14/15, unchanged from 11.1 — same HTML, same AggregateOffer.
D6 — 6/15. Canonical 4 (present, absolute, and identical across probes because both probes received the same document). Cache validators 0: max-age=0 is not greater than zero and there is no validator to revalidate against. llms.txt 0 (404). Sitemap 2. Endpoint 0.
Total: 20 + 0 + 5 + 9 + 14 + 6 = 54 → D, "Partial".
What the pair shows. The markup is not the problem: D1 is 20/20 and D5 is 14/15 on both. The 36-point difference is entirely that everything a caller needs is buried in 91 KB with no machine representation. Moving the agent from 91 226 bytes to 1 490 turned D3.1 from 4 into 14 and D3.2 from 1 into 6 without changing a single fact on the page — the byte reduction lifted the density band, nothing else. This is also why the same publisher's page can be excellent HTML and a D: ARS scores what the caller received.
11.3 021 vs 022 → identical A (98) — the training-neutrality proof
The page: a product page with a gateway. The agent receives 334 bytes of Markdown with three Link values — a typed alternate, a canonical, and a service-desc typed application/json — plus Vary: Accept, Cache-Control: public, max-age=900, an ETag and a Last-Modified. The browser receives 899 bytes of HTML with a complete Product JSON-LD. /llms.txt is 404.
022's robots.txt is User-agent: * / Allow: / plus a Sitemap:. 021's is that file plus one group naming GPTBot, CCBot, Google-Extended and anthropic-ai with Disallow: /.
Derivation (identical for both fixtures):
- D1.1 4 + 2 + 2 = 8. D1.2: the assistant audience matches
*withAllow: /→ not disallowed (3) + clean parse (1) + no orphans (1) + sitemap (1) = 6. The training group is read and not scored. D1.3: 4 of 4 core facts in the HTML → 6. → D1 = 20. - D2.1 machine media type, non-HTML body → 10. D2.2 typed
alternate→ 4. D2.3Vary: Acceptwith D2.1 > 0 → 3. D2.4 compared and agreeing → 3. → D2 = 20. - D3: core 4/4 corroborated (16 units) + 3 corroborated extended (3 units) over
denominator = 16→round½(1900, 16) = ⌊1908 ÷ 16⌋ = 119→ clamped to 100.factsPerKiB100 = ⌊(400 + 75) × 1024 ÷ 334⌋ = ⌊486400 ÷ 334⌋ = 1456→ 100. D3.1 = 14.ratio = ⌊33400 ÷ 8192⌋ = 4→ D3.2 = 6. → D3 = 20. - D4: first offset 0 → 5.
window = max(2048, 66) = 2048; offsets 0, 123, 123, 151 all inside → 100% → 5. → D4 = 10. - D5: all blocks parse → 5.
productrequired properties all satisfied →2 + round½(5 × 4, 4) = 2 + 5 = 7. JSON-LD agrees with the page → 3. → D5 = 15. - D6: canonical present, absolute, and identical on both sides → 4.
max-age=900+ETag→ 3./llms.txt404 → 0. Sitemap → 2.Link rel="service-desc"; type="application/json"→ 4. → D6 = 13.
Total: 20 + 20 + 20 + 10 + 15 + 13 = 98 → A, in both fixtures.
The entire difference between the two results is: policy.audiences.training reads disallow via google-extended in 021 and allow via * in 022; policy.trainingOptOut is true and false; and 021 carries one additional info flag, training-opt-out, whose evidence line reads Effect on this score: none. No point moves. The two missing points in both fixtures are /llms.txt, priced at 2 of 100 for the reason in §5.8.3.
11.4 031 vs 032 → identical C (74) — the stuffing proof
Two stores selling the same kettle from the same theme. The origins are deliberately the same length, so the two HTML bodies differ in exactly one thing: 031 appends a <section class="seo"> carrying keyword spam, a fabricated spec sheet (Material:, Capacity:, Warranty:, Colour:) and a comma-separated keyword tail. 031 is 1 686 bytes; 032 is 995 bytes — 69% larger.
Both serve HTML to both probes, carry a Link: rel="canonical", Cache-Control: public, max-age=600 and an ETag, have a robots.txt with a Sitemap:, and return 404 for /llms.txt and /.well-known/ucp.
Both publish the identical fact set: title @63 (corroborated), description @199 (corr.), availability @334 (corr.), brand @334, currency @334 (corr.), price @334 (corr.), sku @334, returns @788, shipping @788.
Where the two derivations differ, and where they do not:
- Coverage: core 4/4 corroborated → 16 units; extended 5 found, 1 corroborated → 1 unit;
denominator = 16→round½(1700, 16) = ⌊1708 ÷ 16⌋ = 106→ clamped to 100 in both.MaterialandWarrantyare not facts in the vocabulary at all, so they never enter the numerator; and the four core facts were already corroborated, so nothing could have moved this number upward even if the fabricated block had corroborated something real. - Density:
031= ⌊(400 + 125) × 1024 ÷ 1686⌋ = ⌊537600 ÷ 1686⌋ = 318.032= ⌊537600 ÷ 995⌋ = 540. Both are above the 200 threshold, so both takedensityPct = 100. This is saturation doing its job: the extra 691 bytes moved the density number by 222 hundredths and moved the band by nothing. - D3.1 =
⌊(14 × 100 × 100 + 5000) ÷ 10000⌋ = **14**in both. - D3.2:
031ratio = ⌊168600 ÷ 8192⌋ = 20;032ratio = ⌊99500 ÷ 8192⌋ = 12. Both ≤ 100 → 6 in both. - D4.1: first core offset 63 in both → 5. D4.2:
window = 2048in both (⌊1686 ÷ 5⌋ = 337,⌊995 ÷ 5⌋ = 199, both below the 2048 floor); all core offsets inside → 5.
The rest is identical by construction: D1 = 20, D2 = 0 (no negotiation), D5 = 15, D6 = 9 (canonical 4 + cache 3 + llms 0 + sitemap 2 + endpoint 0).
Total: 20 + 0 + 20 + 10 + 15 + 9 = 74 → C, in both fixtures, across all nineteen per-check values, with identical flags and identical recommendations.
Two honest notes. First, what stuffing does change is the cost report: 031's HTML bytes and approximate token estimate are 69% higher, and ARS reports that without pretending the extra bytes were information. Second, this pair does not prove that stuffing is penalised, and it should not: a page that grows far enough falls through the density band on its own, which is what 002 shows at 91 KB.
11.5 018-article-longform → B (77) — the byte reference, and where it stops working
A ~2 400-word magazine feature: 14 284 bytes of HTML, no content negotiation, a single <link rel="alternate" type="application/rss+xml">, Cache-Control: public, max-age=600, an ETag and a Last-Modified, a robots.txt with a Sitemap:, and a /llms.txt that returns 200.
Page kind article at high confidence. Profile: core = title, author, published; extended = updated, section, description; byte reference 40 960.
Facts: title @66, description @131, author @307, published @349, updated @411, section @472 — all six corroborated.
- D1 = 20. 8 + 6 + 6 (all three core facts are in the HTML).
- D2 = 4. D2.1 = 0 (HTML to both probes), D2.2 = 4 (a typed alternate), D2.3 = 0 (gated), D2.4 = 0 (gated).
- D3 = 13. Core 3/3 corroborated → 12 units; extended 3/3 corroborated → 3 units;
denominator = 4 × 3 = 12→round½(1500, 12) = ⌊1506 ÷ 12⌋ = 125→ clamped to 100.factsPerKiB100 = ⌊(300 + 75) × 1024 ÷ 14284⌋ = ⌊384000 ÷ 14284⌋ = 26→ the 20–49 band →densityPct = 50. D3.1= ⌊(14 × 100 × 50 + 5000) ÷ 10000⌋ = ⌊75000 ÷ 10000⌋ = **7**. D3.2:ratio = ⌊1428400 ÷ 40960⌋ = 34→ ≤ 100 → 6. - D4 = 10. First offset 66 → 5.
window = max(2048, ⌊14284 ÷ 5⌋ = 2856) = 2856; offsets 66, 307, 349 all inside → 100% → 5. - D5 = 15. All blocks parse (5);
requiredProperties.article=headline,author,datePublished, all present →2 + 5 = 7; agreement → 3. - D6 = 15. Canonical 4 + cache 3 + llms.txt 2 + sitemap 2 + the RSS alternate as a declared endpoint 4.
Total: 20 + 4 + 13 + 10 + 15 + 15 = 77 → B, "Agent-friendly".
The claim this fixture is for. 14 284 bytes of prose scores 77, three points above 050-byte-identical-replay — a 991-byte product page with flawless JSON-LD and a perfect 20/20 on fact coverage, which scores 74. Long-form content is not structurally capped below short content, because D3.2 charges a page against what its kind reasonably costs. The same 14 284 bytes scored against product's 8 KiB reference would be ratio = ⌊1428400 ÷ 8192⌋ = 174 → 5/6 instead of 6/6, and the gap widens fast: an article stays at 6/6 up to 40 960 bytes, where a product page has already fallen to 3/6; at 002's 91 226 bytes a product scores 1/6 where an article scores 3/6. Absolute byte bands — the design defect the byte reference replaced — would have put a 40 KB article and a 40 KB product page in the same place.
The honest half, which a hostile reviewer should hold this standard to. The byte reference fixes D3.2. It does not reach D3.1, and D3.1 is where a residual long-form penalty lives. Density is (coreFound × 100 + extendedFound × 25) × 1024 ÷ bytes — an absolute facts-per-KiB measure with no page-kind term in it — so six facts across 14 284 bytes lands in the 20–49 band at 50%, and D3.1 pays 7 of 14 at 100% coverage. The band edge is arithmetic, not judgement: with six facts the 20-hundredths threshold sits at 19 200 bytes, so the same piece at 4 000 words scores 74 and drops to C, and stays at 74 until 76 800 bytes. The sentence "the byte reference is what stops long-form content being structurally capped around C" is therefore true at this length and stops being true a thousand words later. That is filed as an open issue (Appendix D), not papered over.
One further note in the same spirit: the single RSS <link rel="alternate"> earns 4 points on D2.2 and 4 on D6.5, even though a section feed is arguably not an alternate representation of this page. Every blog on a common CMS ships that tag.
11.6 020-robots-deliberate-optout → no number at all
Included because a standard that only derives numbers has not shown its most important behaviour.
The page is the same excellent product page that scores 98 (A) in 022. Its robots.txt names four assistant tokens in their own group and disallows them; the file parses with zero malformed lines and zero orphan rules.
The result is not 98, not 40, and specifically not an F:
outcome = { kind: 'opt-out', audience: 'assistant', wellFormed: true }
score = null
grade = null
bandLabel = null
recommendations = []
flags = [ { id: 'assistant-opt-out', severity: 'info', basis: 'measured' } ]All six dimensions are present with their weights and every check reports earned: 0 with a "Not evaluated" reason — the shape is stable, and no sub-score leaks that a ranking could be reconstructed from.
The flag carries the matched group and the matched rule, because a publisher who disagrees with the verdict is entitled to see the line of their own file that produced it. The group reported is the longest matching token, which is RFC 9309 specificity and not a preference.
Two details make this opt-out deliberate rather than accidental, and both are load-bearing: the disallow comes from a group that names assistant tokens, and the file parses cleanly. A blanket User-agent: * / Disallow: / is fixture 023 instead — which is scored, at 92, with D1.2 at 0 and a robots-contradiction warn. Consent has to be specific in both directions.
12. Draft status and calibration
ARS 0.1 ships as a draft, and its bands are not calibrated.
The band edges (90 / 75 / 60 / 40) were chosen from the structure of the allocation rather than from a distribution over real pages. The conformance corpus establishes that the arithmetic is reproducible; it does not establish that a 77 and an 82 describe meaningfully different experiences for a caller.
Consequently:
- Scores produced under ARS 0.1 are declared non-comparable to a future ARS 1.0 unless 1.0 is calibrated first, over a few hundred real pages across all eight kinds, before its bands are frozen.
- Any published time series that spans a MINOR boundary MUST either be re-scored from retained evidence or be labelled as version-segmented and non-comparable. Silently plotting across a MINOR is a misrepresentation of this standard.
- Implementations SHOULD present the band and withhold the integer where the audience is likely to read the integer as precise.
Shipping uncalibrated bands as stable would be worse than shipping them as a draft: the PATCH gate in §2.2 would then force a full re-score weeks after launch, at the exact moment the numbers had started to be quoted.
Appendix A — Ruleset constants
Everything a second implementation needs, in one place. Editing any value here is a spec version change, not a code change.
Weights — 19 checks summing to 100. See the table in §5.1.
Dimension weights — retrievability 20 · machine-representation 20 · fact-coverage 20 · fact-position 10 · structured-data 15 · contract-discovery 15. Each equals the sum of its checks' weights; both are published, and a property test asserts they agree and total 100.
Totals — measuredWeight 64 · heuristicWeight 36 · heuristic ceiling 40.
Threshold tables (flat [threshold, value] pairs):
| Key | Encoding | Value |
|---|---|---|
fact-coverage.fact-units | [corrCore, uncorrCore, corrExt, uncorrExt] | [4, 2, 1, 0] |
fact-coverage.core-denominator-multiplier | scalar | [4] |
fact-coverage.density | AT-LEAST | [200,100, 100,85, 50,70, 20,50, 5,30, 0,15] |
fact-coverage.context-cost | AT-MOST | [100,6, 200,5, 500,3, 1500,1] |
fact-position.first-core-fact-offset | AT-MOST | [512,5, 2048,4, 8192,2, 32768,1] |
fact-position.front-window | AT-LEAST | [90,5, 70,4, 50,3, 25,1] |
fact-position.front-window-size | [floor, divisor] | [2048, 5] |
cost.approx-bytes-per-token | scalar, heuristic | [4] |
Sub-point splits (not members of the hashed ruleset object — see §2.5):
| Check | Split |
|---|---|
| D1.1 | status 2xx 4 · within redirect limit 2 · same final URL 2 |
| D1.2 | assistant allowed 3 · parses clean 1 · no orphan rules 1 · sitemap declared 1 |
| D1.3 | AT-LEAST band [100,6, 75,5, 50,3, 25,1] · <noscript> floor 1 |
| D2.1 / D2.2 / D2.3 / D2.4 | all-or-nothing: 10 · 4 · 3 · 3 |
| D5.1 | clean 5 · partial 3 |
| D5.2 | recognised type 2 · completeness 5 |
| D5.3 | all-or-nothing 3 |
| D6.1 | present 2 · absolute 1 · consistent across probes 1 |
| D6.2 | sane Cache-Control 2 · validator 1 |
| D6.3 | present 1 · spec-shaped 2 |
| D6.4 / D6.5 | all-or-nothing: 2 · 4 |
Probe limits, hashed with everything else — maxBodyBytes 2 097 152 · maxRedirects 3 · timeoutMs 5 000. They are inputs to the score, so they are inputs to the hash: a 3 MiB cap turns a truncated-evidence non-grade into an F, and a 10 s timeout scores pages that a 5 s timeout reports as unreachable.
Bands — see §6.7. Audience tokens — see §7.2. Profiles — see §4.2.
Appendix B — requiredProperties, per page kind (D5.2)
Provenance in §5.7.2. A dot is a property path from the scored node; traversal enters array members.
| Page kind | Required properties |
|---|---|
product | name, offers.price, offers.priceCurrency, offers.availability |
collection | name, numberOfItems, itemListElement |
article | headline, author, datePublished |
place | name, address, openingHours, telephone |
service | name, offers.price, offers.priceCurrency, potentialAction.target |
faq | name, mainEntity, mainEntity.acceptedAnswer |
document | name, dateModified, publisher |
unknown | name |
unknown requires only name, deliberately: §4.4 scores unknown against "any recognised schema.org type is present", which requires having a type but cannot require properties of a kind we did not identify.
Appendix C — The action lexicon
Matched case-insensitively against normalised anchor text, <form> context, or potentialAction.name, with whitespace collapsed. Pinned; sorted for readability; order is not significant.
add to bag · add to basket · add to cart · apply · apply now · book · book now · buy · buy now · call now · checkout · contact · contact us · donate · download · email us · enroll · get a quote · get quote · get started · join · make an appointment · order · order now · purchase · register · request · request a demo · request a quote · reserve · schedule · sign up · signup · start · start free trial · subscribe
Appendix D — Known limitations and open issues in 0.1
Published as part of the standard, because a standard that hides its own defects is asking to be trusted rather than checked. Each of these is visible in a named fixture or in the reference implementation, and each is a candidate for 0.2.
| # | Issue | Where it is visible |
|---|---|---|
| D-1 | corpusHash is not yet a corpus digest. It is the SHA-256 of the string ars-0.1-corpus-unfrozen until the corpus freezes | §2.4.3 |
| D-2 | SUBPOINTS, the classification tables and the extraction tables are not covered by rulesetHash. They are pinned by the corpus instead | §2.5 |
| D-3 | D5.2 walks JSON-LD only. Microdata and RDFa satisfy D5.1 but are not traversed for required properties, so a microdata-only page cannot exceed 5/15 on D5 | §5.7.2 |
| D-4 | A declared paywall is flagged, not exempted. The rule that the gated part is exempt from coverage and position needs hasPart + cssSelector support and is not implemented | 046-paywall-declared |
| D-5 | The ≥30 s delay on the parity-confirmation probe is not verifiable by the scorer. The evidence bundle has no per-probe timestamp, so the delay is a contract on the probe | 030-accept-conditional-divergence |
| D-6 | "Pricing on request" satisfies D3.1 and cannot satisfy D5.2. The service row of requiredProperties asks for offers.price, so the same page is rewarded for candour by one check and charged two points for it by another. Closing this means editing the table, which is a MINOR | 013-service-pricing |
| D-7 | A nested Brand node's name can win the title tie-break and manufacture a structured-data-divergence. A correct install is not flagged in the fixtures; it is not yet true in general | 038, and the same tie-break in 042, 046, 048 |
| D-8 | The money parser does not know the fullwidth yen sign (U+FFE5), which is the only yen sign a Shift_JIS page can contain. A universality claim with a gap in it | 048-shift-jis-charset |
| D-9 | D3.1's density term has no page-kind component, so long-form content still pays a coverage penalty that the byte reference does not reach | §11.5 |
| D-10 | Vantage variance is not enforced. A site that serves well only to known scanner egress is not detected in 0.1 | §9 |
| D-11 | Site-level scoring is informative only and appears on no published surface | §1.1 |
| D-12 | An earlier draft of the design describes a truncated capture as scored with truncated: true surfaced, while the outcome vocabulary lists truncated-evidence among the non-grades. This standard and the implementation follow the non-grade reading | §3.3, 043-body-over-cap-truncated |
Appendix E — Colophon
ARS 0.1 is published by Rebilder. The conflict of interest is disclosed in §0.1 and is the reason every threshold in this document is a number rather than a description.
Corrections, disagreements with a weight, and independent implementations are all welcome, and the corpus is the place to argue: a fixture that this standard scores wrongly is a better argument than a paragraph, and it is one we have to answer.