Training and retrieval are different problems
Brand teams often treat an assistant’s wrong answer as one problem. It is two, with different time constants and different remedies, and the useful diagnostic is whether the answer changes when the model is allowed to browse.
| Training knowledge | Retrieved knowledge | |
|---|---|---|
| Where it came from | A corpus frozen at some past date | A live fetch of a URL |
| How stale it can be | Arbitrarily | As stale as your page |
| Can you correct it | Not directly | Yes, by changing the page |
| How long a fix takes | A model generation, if ever | The next fetch |
| Who controls it | The model provider | You |
This is why the practical brand strategy is to make retrieval win. A model that can cheaply fetch a current, unambiguous page about you has less reason to fall back on whatever it half-remembers, and that is the entire mechanism available to you.
Three properties decide whether your page is used once it is fetched, and all three are ordinary engineering rather than content strategy.
- Reachable. No bot rule, CDN filter or challenge page between the agent and the content. Most blocks on assistants were inherited from an old anti-scraping policy rather than chosen.
- Complete. The facts a person would ask about are present in the text a parser sees, not rendered client-side and not implied by an image.
- Cheap. On the reference document capture, the same page fell from 13,581 bytes to 1,412 and the first fact moved from character 10,152 to line 8. Cheap pages survive truncation; expensive ones get summarised badly.
What to report internally
Report the measurements, label the samples, and resist converting the two into one number. A single blended "visibility score" combining exact server data with sampled prompt results is the metric most likely to be quietly wrong in a quarterly review.
- Baseline the templates. One readability score per page kind, taken before any change.
- Report fetches and answer rate monthly. Exact, from your own logs, and defensible.
- Track the miss log as a backlog, not as a metric. It is a to-do list that happens to be ranked by demand.
- If you sample assistant answers, say so on the slide. It is indicative, and treating it as a target invites optimising for a number nobody controls.