/.well-known/rebilder-optout, version 1. Status: published. Stable — version 1 will not change meaning.
0. What this is, and what it is not
This document specifies a file a domain owner publishes on their own origin to tell Rebilder: do not publish anything about this domain.
It is not a robots file and it does not replace one. The two govern different things, and the split is the whole point:
| Governs | Mechanism | |
|---|---|---|
robots.txt | Whether we measure — whether our fetcher may request your pages at all | User-agent: rebilder-ars + Disallow: |
| This file | Whether we publish — whether your domain may appear in a public index, ranking, or badge | Index: deny |
You can allow measurement and refuse publication. You can refuse both. Refusing publication does not require you to block a fetcher you may otherwise want running.
0.1 This file can only ever subtract
Rebilder publishes nothing about a named third party without verified domain-owner opt-in (a DNS TXT record proving control, via https://rebilder.com/readable/verify). That requirement stands on its own. This file exists so that an owner who wants to state a refusal once, in advance, without ever visiting rebilder.com can do so.
A consequence worth stating plainly: the absence of this file is not consent, and a Index: allow in it is not consent either — see §5.
1. Location and scope
The file is served at:
https://<host>/.well-known/rebilder-optout
Scope is the host it was fetched from, and every host below it. A file at example.com covers example.com, shop.example.com, and a.b.example.com. A file at shop.example.com covers shop.example.com and below, and says nothing about example.com.
When both a registrable domain and a more specific host publish a file, the more restrictive answer wins. Index: deny anywhere in the chain is a deny. There is no override direction; a parent cannot re-permit what a child refused, and a child cannot re-permit what a parent refused.
2. Syntax
A UTF-8 text file. One directive per line, Name: value, in the shape of an HTTP header field.
# Comments start with a hash and run to end of line. Version: 1 Index: deny Contact: mailto:webmaster@example.com
Rules:
- Field names are case-insensitive.
index,IndexandINDEXare one
field.
- Values are trimmed of leading and trailing whitespace. Enumerated values
(deny, allow) are case-insensitive.
- Blank lines and comment lines are ignored. A
#only starts a comment at
the beginning of a line, after optional whitespace — it is not stripped from inside a value, because a URL may contain a fragment.
- Unknown field names are ignored. This is the forward-compatibility hinge:
a version 2 parser may understand fields a version 1 parser skips, and a version 1 file stays valid.
- A repeated field is an error, not a last-one-wins. Two
Index:lines
disagreeing is exactly the case where guessing is wrong (§4).
- Line endings may be LF or CRLF.
2.1 Limits
A parser MUST stop reading at whichever comes first:
- 8192 bytes, or
- 100 lines.
A file longer than either limit is truncated at the limit, and the truncated content is parsed as if it were the whole file. These are defensive limits against a misconfigured origin serving a large document at this path; a real opt-out file is under ten lines.
3. Directives
3.1 Version: — required
Version: 1
An integer. A file without a parseable, supported Version: is not an opt-out document (§5.1) — it is treated as absent, not as a refusal.
A file declaring a version this parser does not support is a deny (§4). The owner is deliberately speaking to us in a dialect we cannot read, and the safe reading of an unreadable refusal is that it is a refusal.
3.2 Index: — required
Index: deny
| Value | Meaning |
|---|---|
deny | Do not publish anything about this host in a public index, ranking, badge, or comparison. |
allow | The owner does not object to publication. Not consent — see §5.2. |
Any other value, or a missing Index: in an otherwise recognised document, is a deny (§4).
3.3 Contact: — optional
Contact: mailto:webmaster@example.com
A single URI, mailto: or https:. Where to reach the operator about this file. We do not use it for sales contact — an address published to manage a publication refusal is never a marketing lead, because contact conditioned on a removal is the one form of outreach that stays forbidden (CLAUDE.md Hard Rule 9, as amended Aug 2026) — and we do not store it beyond the crawl record.
3.4 There are no other directives
No paths, no wildcards, no user-agent groups, no expiry. The file answers one question about one host tree. Anything more expressive is a second robots.txt with second-system semantics, and the failure mode of a misunderstood publication refusal is that we publish something someone refused.
4. Failure is a deny
Every ambiguity inside a recognised document resolves to deny:
| Condition | Result |
|---|---|
Unsupported Version: | deny |
Missing Index: | deny |
Index: with an unrecognised value | deny |
| Any field repeated | deny |
| Truncation limit hit mid-document | parse what was read; normal rules apply |
And every ambiguity about fetching the document resolves to "do not publish this round":
| Fetch outcome | Result |
|---|---|
200 with a recognised document | Parse it (§3, §4) |
200 with anything unrecognisable | no-document (§5.1) |
404, 410 | no-document |
429, 5xx, timeout, TLS failure, DNS failure | unknown — do not publish this host until a later fetch resolves |
| Redirect to another origin | Not followed. no-document. |
| Redirect within the same origin | Followed, up to 3 hops |
unknown is deliberately not no-document. A publication decision taken while the operator's stated policy is unreadable is a decision taken without it.
5. What this file cannot do
5.1 It cannot be accidental
A large fraction of sites answer 200 with their homepage for any unknown path. If that were read as a malformed opt-out file, and malformed meant deny, a soft-404 would silently remove domains whose owners had verified and consented.
So: a response is an opt-out document only if it is recognisable as one. It must carry a parseable Version: field within the limits of §2.1, and it must not be HTML (a body whose first non-whitespace bytes are <, or a Content-Type of text/html, is never a document). Anything else is no-document — no signal in either direction.
5.2 It cannot grant consent
Index: allow is a signal, not an authorisation. Publication requires the DNS TXT verification flow, and it will continue to.
The reason is the threat model. Control of /.well-known/ on an origin is a weaker proof than control of DNS: shared hosting, a misconfigured CDN rule, a path-prefixed tenant, or an unclaimed subdomain can all let someone who is not the owner write a file there. The asymmetry decides it — a forged deny costs us a listing we were not entitled to anyway, while a forged allow would publish a domain whose owner never consented. So deny is honoured unconditionally and allow is honoured never.
5.3 It cannot be bought back
Removal from anything we have published is free, self-serve, permanent, and never purchasable. This file is one of the routes to it. There is no route in the other direction that costs money.
6. Crawler behaviour
- The index crawler fetches this file once per origin per 24 hours, before
anything else, and caches the result for that period. It is one request; it is not part of the per-URL scan budget documented at https://rebilder.com/bots.
- The scanner does not fetch it. A user-initiated scan produces a
session-private result that is never published, so a publication refusal has nothing to govern. robots.txt is what governs the scanner.
- The publisher re-reads the decision at publication time. A
denythat
appears between crawl and publish is honoured; the window is the cache TTL above.
7. Examples
Refuse publication, allow measurement. The common case.
Version: 1 Index: deny
Refuse publication and measurement. This file plus a robots.txt group.
Version: 1 Index: deny Contact: mailto:security@example.com
User-agent: rebilder-ars Disallow: /
State no objection. Legal, and does nothing on its own (§5.2).
Version: 1 Index: allow
8. Reference implementation
rebilder.com serves its own file at https://rebilder.com/.well-known/rebilder-optout. It is a live example of the format, and it is the file our own domain is governed by.
The parser is apps/rebilder-api/lib/optout.ts in the Rebilder monorepo, and the examples in §7 above are executed as test fixtures — this document is read by apps/rebilder-api/tests/unit/optout.test.ts, which parses every fenced block in it. A change to an example here that the parser disagrees with fails the build.