Disclosure
Our scanner and crawler
If you found us in an access log, this is the page our User-Agent points at.
- Visit
- Request
- Response
User-Agent
rebilder-ars/0.2 (+https://rebilder.com/bots)
We operate one fetcher. It measures how readable a page is for AI agents, under the published Agent Readability Spec 0.2.0. It does not index your content, train on it, or store your response bodies for publication.
Opting out
Name rebilder-ars in robots.txt
One group, two lines, and we stop. We publish a token specifically so you can refuse us without refusing assistants.
User-agent: rebilder-ars Disallow: /
Only a group that names us counts
A blanket User-agent: * disallow does not trigger the scanner refusal. That is deliberate: blanket disallows are often accidental, so we only treat a rule that names us as a refusal.
A refusal produces no score
A scan of a domain that has named us produces no score and no letter grade, just unscored / robots-disallow-scanner. We do not fall back to a cached number, and we never publish a refusal as a negative.
Refusing assistants is a separate thing
A deliberate, well-formed disallow that names assistant crawlers is recorded as an opt-out, not a failure. It carries no letter grade and is never listed or ranked.
Blocking training crawlers costs nothing
Blocking model-training crawlers while allowing assistant fetches scores identically to a fully open site. Two published conformance fixtures prove it: the same page with and without a training-crawler disallow, both scoring 98. The only difference is an informational flag with a stated score effect of "none".
Removal from anything we published
robots.txt stops future measurement. If a domain has an entry in the public index, removal is separate, free, self-serve, permanent, and never purchasable. An unverified removal request still triggers suppression pending review.
Refusing publication without blocking anything
robots.txt governs whether we measure. A file at /.well-known/rebilder-optout governs whether we publish: whether your domain may appear in any index, ranking or badge. They are separate on purpose: refusing to be listed should not require blocking a fetcher you may otherwise want running.
# https://rebilder.com/spec/optout Version: 1 Index: deny
The format is published at /spec/optout, and rebilder.com serves its own file at the same path. A malformed file, a 5xx, or a timeout counts as a refusal: when in doubt, we do not publish.
One request per origin per day, before anything else, cached for 24 hours. It belongs to the index crawler and is not part of the scan budget below: a user-initiated scan produces a private result that is never published, so a publication refusal has nothing to govern there.
Request budget
What a scan actually costs you
Two requests per URL, three per origin, cached. A cold single-URL scan is five requests; a warm origin is two.
| Request | When | Why |
|---|---|---|
| /robots.txt | First, always | Policy. Nothing else is fetched until this resolves. |
| the target URL | Agent probe | Accept prefers a machine representation. This is what a caller that wants facts receives. |
| the target URL | Browser control | Accept prefers HTML. Identical to the agent probe in every other respect: same User-Agent, no cookies. |
| /llms.txt | Per origin | Scored by D6.3. |
| /.well-known/ucp | Per origin | Scored by D6.5. |
| the target URL | Only on detected divergence, ≥30s later | Parity confirmation. Skipped entirely on the overwhelming majority of scans. |
Rate limits we impose on ourselves
- At most one concurrent request per host, and a small global concurrency limit.
- At least one second between requests to the same host.
- A scan of one URL costs a small fixed number of requests: the page itself, twice (one HTML, one markdown), plus robots.txt, llms.txt and /.well-known/ucp. The public scanner caches the whole result for 24 hours per URL, so a repeated scan of the same URL costs nothing.
- A soft per-run cap with an explicit override, so a runaway loop stops on its own rather than on your rate limiter.
- The probe waits at least one second between requests to the same host, always. Crawl-delay is not read: the fixed interval is the guarantee, and it is never scored, so nothing in your robots.txt can hurt your grade.
- A 429 ends the scan for that URL rather than retrying it, and the page is left unscored. A persistent 5xx on robots.txt stops the scan; a single hiccup does not.
What we never fetch
- We do not guess at /sitemap.xml, /mcp, /acp, /openapi.json, or any other path. The three probed files above are the whole list.
- We never execute JavaScript, so we make no subresource requests at all: no images, no scripts, no stylesheets, no XHR.
- We send no cookies and store none.
- We never send a User-Agent belonging to Googlebot, ChatGPT, Claude, or any other operator. Every request identifies as ours.
The two policies
Two kinds of fetch, two rules
Index crawling and one-off checks follow different rules. Both are published here.
Published or permanent artifacts
Full robots.txt compliance
Anything that results in a stored, published, or permanent artifact (an index entry, a badge, a row in an aggregate) obeys robots.txt, Crawl-delay, 429/Retry-After, and the published opt-out endpoint. This is the index crawler, and it is the strict policy with no carve-outs.
Session-private, user-initiated checks
The PageSpeed Insights precedent
A person pasting one URL into a tool and seeing the result themselves is a one-off, user-initiated fetch on that person's behalf, in the same category as a browser developer tool or a page-speed checker. Those results are shown to the requesting session and are not published, indexed, or aggregated.
Even so, we never rely on the second policy. ARS 0.2.0 publishes nothing about a non-consenting domain: named entries require verified opt-in, and aggregates require 25 independent domains and name nobody.
Naming rebilder-ars in robots.txt stops both, including user-initiated scans from our public vantage. The one exception: a site’s own operator can always scan their own domain.
Identity
How to verify a request is really us
The User-Agent is stable
Every request carries rebilder-ars/0.2 (+https://rebilder.com/bots), and the token rebilder-ars is what robots.txt matches on. Matching is case-insensitive.
Published IP rangesPlanned
We do not publish IP ranges today. The scanner runs on shared serverless infrastructure, so its egress addresses are not ours to publish.
Published ranges will ship when the crawler moves to fixed egress. Until then the User-Agent is the only identifier, and it can be spoofed, so treat it accordingly.