Disclosure
Our scanner and crawler
If you found us in an access log, this is the page our User-Agent points at.
User-Agent
rebilder-ars/0.1 (+https://rebilder.com/bots)
We operate one fetcher. It measures how a page reads to a caller that wants facts rather than a rendered document, under the published Agent Readability Spec 0.1.0. It does not index your content, it does not train anything on it, and it does not store your response bodies for publication.
Opting out
Name rebilder-ars in robots.txt
One group, two lines, and we stop. We publish a token specifically so you can refuse us without refusing assistants.
User-agent: rebilder-ars Disallow: /
Only a group that names us counts
A blanket User-agent: * disallow does not trigger the scanner refusal. That is deliberate: blanket disallows are frequently accidental, and treating one as consent to stop measuring would let a misconfiguration decide something the operator never decided.
We obey it, and we say so
A scan of a domain that has named us produces no score and no letter grade. It produces a stated non-result: unscored / robots-disallow-scanner. We do not fall back to a cached number and we do not publish the fact that you refused as a negative.
Refusing assistants is a separate thing
A deliberate, well-formed disallow that names assistant crawlers is recorded as an opt-out, not a failure. It carries no letter grade and is never listed or ranked. A choice is not a low score.
Blocking training crawlers costs nothing
Blocking model-training crawlers while allowing assistant fetches scores identically to a fully open site. This is not a policy statement we ask you to believe — two conformance fixtures are the same page with and without a training-crawler disallow, and both score 98. The difference between their results is one informational flag whose stated effect on the score is "none".
Removal from anything we published
robots.txt stops future measurement. If a domain has an entry in the public index, removal is separate, free, self-serve, permanent, and never purchasable — and an unverified removal request still triggers suppression pending review.
/.well-known/rebilder-optoutPlanned
The index crawler will additionally honour a published opt-out endpoint at /.well-known/rebilder-optout. The crawler is not running yet, so the endpoint is not yet consulted; robots.txt is the mechanism that works today.
Request budget
What a scan actually costs you
Two requests per URL, three per origin, cached. A cold single-URL scan is five requests; a warm origin is two.
| Request | When | Why |
|---|---|---|
| /robots.txt | First, always | Policy. Nothing else is fetched until this resolves. |
| the target URL | Agent probe | Accept prefers a machine representation. This is what a caller that wants facts receives. |
| the target URL | Browser control | Accept prefers HTML. Identical to the agent probe in every other respect — same User-Agent, no cookies. |
| /llms.txt | Per origin | Scored by D6.3. |
| /.well-known/ucp | Per origin | Scored by D6.5. |
| the target URL | Only on detected divergence, ≥30s later | Parity confirmation. Skipped entirely on the overwhelming majority of scans. |
Rate limits we impose on ourselves
- At most one concurrent request per host, and a small global concurrency limit.
- At least one second between requests to the same host.
- Origin-level fetches (robots.txt, llms.txt, /.well-known/ucp) are cached for at least an hour, and for 24 hours in the public scanner. A warm origin costs two requests per URL.
- A soft per-run cap with an explicit override, so a runaway loop stops on its own rather than on your rate limiter.
- Crawl-delay is honoured by the probe. It is never scored — waiting longer is not worse markup.
- 429 and Retry-After are honoured. A persistent 5xx on robots.txt stops the scan; a single hiccup does not.
What we never fetch
- We do not guess at /sitemap.xml, /mcp, /acp, /openapi.json, or any other path. The three probed files above are the whole list.
- We never execute JavaScript, so we make no subresource requests at all: no images, no scripts, no stylesheets, no XHR.
- We send no cookies and store none.
- We never send a User-Agent belonging to Googlebot, ChatGPT, Claude, or any other operator. We imitate agent intent, never agent identity — every request we make is honestly attributable to us.
The two policies
Not every fetch is the same kind of fetch
We publish both rules rather than the convenient one.
Published or permanent artifacts
Full robots.txt compliance
Anything that results in a stored, published, or permanent artifact — an index entry, a badge, a row in an aggregate — obeys robots.txt, Crawl-delay, 429/Retry-After, and the published opt-out endpoint. This is the index crawler, and it is the strict policy with no carve-outs.
Session-private, user-initiated checks
The PageSpeed Insights precedent
A person pasting one URL into a tool and seeing the result themselves is a one-off, user-initiated fetch on that person's behalf — the same category as a browser developer tool or a page-speed checker. Those results are shown to the requesting session and are not published, indexed, or aggregated.
We think the second policy is defensible, and we have arranged things so that we never have to rely on it. ARS 0.1.0 publishes nothing about a non-consenting domain: named entries require verified opt-in, and aggregates require 25 independent domains and name nobody. So the aggressive reading of the precedent is never load-bearing, and if you disagree with it, the disagreement costs you nothing.
Naming rebilder-ars in robots.txt stops both, including the user-initiated one, from a public vantage. The single exception is a site's own operator scanning their own origin — the owner can consent for their own domain, and telling the most careful publishers that they may not look at their own site would be a product regression dressed as rigour.
Identity
How to verify a request is really us
The User-Agent is stable
Every request carries rebilder-ars/0.1 (+https://rebilder.com/bots), and the token rebilder-ars is what robots.txt matches on. Matching is case-insensitive.
Published IP rangesPlanned
We do not publish IP ranges yet, and we would rather say that than publish a range we cannot hold. The scanner currently runs on shared serverless infrastructure whose egress addresses are not ours to publish.
Fixed egress with published ranges lands with the crawler, in the same change that moves it out of a serverless function. Until then the User-Agent is the only identifier we ask you to trust, and you should treat it accordingly.