# [AI search optimization: the definitive guide](https://rebilder.com/learn/ai-search-optimization)

> AI search optimization end to end: how assistants actually retrieve pages, the four levers you control, measured before/after numbers, and how to track it.

- **Updated:** 2026-08-16

AI search optimization is the work of making a page legible to the software that now reads it first. An assistant fetches your URL, parses what comes back under a token budget, and either states your facts accurately or moves on to a site where it can. That fetch is the new impression.

This guide is the whole discipline in one place: what the term means, how retrieval actually works, the four levers you control, what the change is worth in measured bytes and tokens, and how to tell whether any of it worked. It is written for any site, not only stores. The measurements below include a services page precisely because most of the web is not a product page.

## What AI search optimization actually means

AI search optimization is the practice of getting your content retrieved, understood, and cited by AI assistants and the agents acting on their users’ behalf. It replaces the ranking question with a legibility question. Nobody is competing for a blue link position inside a chat answer; there is no position. There is only whether the assistant could extract your facts cleanly enough to repeat them with confidence.

That shift matters because the failure mode is silent. A page that ranks badly still shows up in a report. A page an assistant could not parse produces nothing at all: no impression, no click, no line in your analytics. The visit happened. Your tooling did not see it, because an agent fetch runs no JavaScript and fires no page-view beacon.

> **The one-read test** Could a competent assistant, given only the bytes your server returns for a page, state the three facts a buyer needs from it in a single read? For most sites the answer is no. The facts are in the response, buried under theme markup, widgets, and scripts that only a browser was ever going to run.

## AEO, GEO, LLM SEO, AI SEO: one job, four names

The vocabulary is unsettled, which makes the field look more fragmented than it is. Four terms are in circulation and they describe overlapping slices of the same job:

| Term | What it emphasises | Where it differs |
| --- | --- | --- |
| AI search optimization | The whole discipline: retrieval, extraction, citation | The broadest of the four; the others fit inside it |
| Answer engine optimization (AEO) | Being the source an answer is built from | Framed around the answer, so it under-weights the fetch that precedes it |
| Generative engine optimization (GEO) | Appearing inside generated responses | Often used for content tactics only, with no serving-layer work |
| LLM SEO / AI SEO | Classic SEO habits pointed at assistants | Useful shorthand; risks implying keyword tactics are the mechanism |

Pick whichever term your team already uses. The work does not change: make the page cheap to fetch, make the facts easy to find in it, and make sure nothing in your stack blocks the fetch in the first place. Everything below is that work in order of payoff.

## How AI search retrieval actually works

There are two paths by which your content reaches an assistant’s answer, and only one of them is under your control this quarter.

1. **Training and pre-built indexes.** Your pages were crawled at some point in the past, and something about them survives in weights or in a vendor’s index. You influence this the way you always influenced search: publish, get linked, be crawlable. The feedback loop is measured in quarters, and you cannot inspect it.
2. **Live retrieval.** The assistant fetches your URL right now, mid-conversation, because the user asked something it cannot answer from memory. It reads what your server returns in that moment. The feedback loop is one HTTP request long, and you control every byte of it.

Live retrieval is where AI search optimization has leverage. It is also where most sites fail, because the response was designed for a browser. An assistant asking for your pricing page receives the same eighty kilobytes of navigation, cookie banner, analytics snippets, and third-party widgets a human browser would, and it has to find your fees inside that under a context budget it is also spending on the rest of the conversation.

The mechanism that fixes this is not new and not proprietary: HTTP content negotiation. An agent sends `Accept: text/markdown`, and a server that understands the header answers with clean structured text instead of a rendered page. The header has been in the HTTP spec for decades. What is new is that clients finally send it and almost no servers answer it.

- **~4.2x**: Content negotiation vs llms.txt, accurate retrieval
- **~38%**: Conversion lift, AI-referred visitors vs search
- **~7x**: Cyber Week 2025 sales growth with agent integration

That first figure is the one to read twice. A 300k-domain study in mid-2026 measured content negotiation as roughly 4.2x more effective than llms.txt for accurate retrieval, and found llms.txt alone produced no measurable citation lift. Publish an llms.txt anyway, because it is a useful index of what you have. Do not mistake it for the mechanism.

## What classic SEO still does, and where it stops

AI search optimization does not replace SEO. It sits on top of it, and a site with broken fundamentals cannot skip to the new layer. Everything in the first column below is still load-bearing.

| Still works exactly as before | Does nothing for an assistant |
| --- | --- |
| Being crawlable and returning fast, stable responses | Keyword density and heading stuffing |
| Clear titles that state the subject of the page | Above-the-fold layout and visual hierarchy |
| Internal links that let a crawler reach every page | Interstitials, carousels, and modals |
| Accurate structured data where a spec exists | Anything that only exists after JavaScript runs |
| Topical authority and citations from real sources | Word-count padding to hit a target length |

The right column is worth dwelling on. Three of those five items are not neutral for agents, they are actively expensive: every kilobyte of layout markup is context the assistant spends before it reaches your facts, and anything rendered client-side is simply absent from what a fetch returns.

> **The cloaking line, stated plainly** Serving markdown to an agent and HTML to a browser is safe when the substance is identical. Different prices, availability, or claims by requester is cloaking, and it is penalised. Known search crawlers should always receive your canonical HTML. Same substance, different format is the defensible position; anything else is not.

## The four levers you actually control

Strip the tactics lists down and four things move the outcome. They are ordered by payoff, and the first one is free to check.

1. **Access.** Confirm nothing in your stack blocks the fetch. Bot rules in a CDN, a WAF challenge, an aggressive `robots.txt`, or a login wall in front of public content will each turn your site into a page that does not exist. This is the cheapest failure to have and the most common one; check it before optimising anything.
2. **Format.** Answer `Accept: text/markdown` with clean structured text. This is the single highest-leverage change, and it is middleware rather than a redesign: your human site does not change at all. [Rebilder’s agent gateway](/docs/quickstart) does it free, with adapters for Next.js, Shopify, Express, Fastify, and Cloudflare Workers.
3. **Facts.** Front-load the values an assistant is actually asked for. Price, availability, hours, fees, shipping, returns, eligibility, contact. The measurement further down shows why position matters as much as presence: a fact at character offset ten thousand, inside a sidebar table, is a fact the assistant may never reach.
4. **Index.** Publish an `llms.txt` so an agent can discover what you have without crawling you blind, and keep it generated from the same source of truth as your pages. Build one in a minute with the [free generator](/tools/llms-txt-generator), let the gateway [emit it](/docs/llms-txt) automatically, or read the [full llms.txt guide](/learn/llms-txt) first, including the evidence on what it does and does not do.

Notice what is not on the list. There is no content-volume lever, no schema-everything lever, and no submit-to-an-AI-directory lever. The four above are the ones with a mechanism you can point at, and the first two are serving decisions rather than content projects.

## What the change is worth, measured

Two reference captures answer the same question against two different kinds of page: what does an agent receive, before and after a gateway answers the `Accept` header? Both are deterministic, generated from committed fixtures with no live network, and re-verified on every build.

The first is a commerce product page, which is the favourable case: it has a price, a buy button, and structured markup much of the ecosystem already parses.

| Commerce capture (a PDP) | Before: full page HTML | After: gateway markdown |
| --- | --- | --- |
| Bytes | 91,226 | 1,315 (−98.6%) |
| Approx tokens | ~22,789 | ~328 |
| First price location | Character offset 23,185, inside a JSON blob | Line 4 |

Roughly **69× less context** for the same buying facts. The second capture is a physiotherapy clinic’s services page: no price, no product markup, none of the ecosystem’s existing assumptions to lean on. It is the case that tests whether this generalises.

| Document capture (a services page) | Before: full page HTML | After: gateway markdown |
| --- | --- | --- |
| Bytes | 13,581 | 1,412 (−89.6%) |
| Approx tokens | 3,388 | 349 (9.7× fewer) |
| Opening hours location | Character offset 10,152, inside a sidebar table | Line 8 |

> **Why two numbers and not one average** A product page and a services page carry different amounts of chrome, so the reductions differ and averaging them would invent a figure neither capture supports. Both are cited on their own terms wherever they appear.

The size reduction is the headline; the ordering is the substance. In the clinic capture the fee list sits at character offset 10,509 of the HTML, and the page’s own structured data describes opening hours but carries no fees and no accepted insurers at all. An assistant that trusts the structured data gives a confidently incomplete answer, which is worse for the clinic than no answer.

What the clinic’s page looks like after (abridged)

```markdown
# [Fernwood Physiotherapy](https://fernwood-physio.example/clinic/bristol-fernwood)

> Chartered physiotherapy clinic in Bristol treating sports injury, back and neck pain, and post-operative rehabilitation.

- **Address:** 14 Fernwood Road, Bristol BS6 7NN
- **Phone:** 0117 496 0188
- **Opening hours:** Monday-Thursday 7:30am - 8:00pm, Friday 7:30am - 6:00pm
- **Initial assessment (45 min):** £78.00
- **Follow-up (30 min):** £56.00
```

Every question a patient asks before booking is answerable from the first eight lines. That is the whole objective of AI search optimization, expressed as a file.

## How to measure AI search optimization

Standard analytics cannot see this traffic. An agent fetch runs no JavaScript, so it never fires a page-view beacon; at best it registers as a bounce, and usually it registers as nothing. Measuring AI search optimization means observing requests rather than page views. Four numbers are worth tracking, in this order:

- **Agent visits.** How many requests came from assistants and agents, split by platform. This is the impression count of the agentic web, and until you have it every other number is a guess. [Detection is a fixture-backed classification](/docs/classification) of the request, not a guess from the user-agent string alone.
- **Answer rate.** What share of agent requests your site actually answered with structured content rather than falling back to HTML. This is the number that moves when you ship format work, and it moves the same week.
- **The miss log.** The specific URLs agents asked for that your site could not answer. This is the highest-value list you will ever get from this work, because each row is a page an assistant wanted and did not get. It needs no other customer’s data to be useful, which is why it is worth reading on day one.
- **The citation funnel.** Visits, then citations, then referred humans, then conversions. The last step is where AI search optimization stops being a technical metric: AI-referred visitors convert ~38% higher than search visitors, so a referred session is worth more than a search session, not less.

One measurement discipline is worth borrowing from the start: score before you change anything. A readability score taken on the current site is the baseline the next three months of work is measured against, and taking it afterwards is the most common way teams end up unable to prove the change was worth doing.

## The playbook, in order

A week of work, ordered so the cheapest checks come first and nothing later depends on a decision you have not made yet.

1. **See what an agent gets today.** Fetch one important page twice: `curl -H 'Accept: text/markdown' https://your-site/page` and then plain. Read both. Most teams stop being sceptical about this work at exactly this point.
2. **Unblock the fetch.** Review `robots.txt`, CDN bot rules, and WAF challenge settings for anything that turns an assistant away from public pages. Decide deliberately which agents you serve rather than discovering the answer by accident.
3. **Install content negotiation.** Middleware that detects agents and answers with markdown rendered from your own data, passing every human and search crawler straight through. The [quickstart](/docs/quickstart) is a single install and a few lines of wiring.
4. **Front-load the facts.** For each page type, list the three to five values a user would ask an assistant about it, and make sure every one is in the first screenful of the response rather than recoverable only by parsing your markup.
5. **Publish an index.** Ship `llms.txt` generated from the same source of truth, so it cannot drift from the pages it points at.
6. **Instrument, then iterate.** Watch agent visits, answer rate, and the miss log weekly. The miss log is the backlog: it tells you which page to fix next, in the order the market is asking for them.
7. **Plan for protocols without waiting for them.** UCP, ACP, and MCP endpoints run off the same source-of-truth wiring as your markdown. Get the wiring right and adding an endpoint is a config change rather than a project.

## Five mistakes that cost the most

- **Treating llms.txt as the strategy.** It is an index. The mid-2026 300k-domain study found no measurable citation lift from llms.txt alone, against roughly 4.2x for content negotiation. Ship both; know which one is doing the work.
- **Blocking the agents you want.** A `robots.txt` rule or a CDN bot filter added years ago against scrapers now blocks the assistants your customers ask. Audit the rules deliberately instead of inheriting them.
- **Rendering the facts client-side.** A price or an opening-hours table that only appears after JavaScript runs does not exist in the response an agent reads. This is the single most common reason a page scores badly with no visible problem in a browser.
- **Varying substance by requester.** Serving a different price or claim to an agent than to a browser is cloaking. It also destroys the thing you are optimising for, because an assistant that cites a price your site does not honour stops citing your site.
- **Measuring with page-view analytics.** Agent fetches run no JavaScript. If your only instrument is a page-view script, your AI search optimization programme is being evaluated against a number that structurally cannot contain its results.

## What is AI search optimization?

AI search optimization is the practice of making your pages retrievable, parseable, and citable by AI assistants and the agents acting for their users. In practice it means making sure the fetch is not blocked, answering `Accept: text/markdown` with clean structured text, front-loading the facts an assistant is asked for, and publishing an llms.txt index.

## Is AI search optimization different from SEO?

It builds on SEO rather than replacing it. Crawlability, fast stable responses, clear titles, internal linking, and accurate structured data all still matter. What changes is that the reader is a parser with a token budget instead of a person with a screen, so layout, visual hierarchy, and anything rendered client-side stop counting, and the cost of your markup starts counting against you.

## What is the difference between AEO, GEO, and AI search optimization?

They describe overlapping slices of one job. Answer engine optimization frames it around being the source an answer is built from; generative engine optimization frames it around appearing inside generated responses, often as content tactics only; AI search optimization is the broadest of the three and includes the serving layer, which is where the mechanism actually lives.

## How do I know if AI agents can read my site?

Fetch one page twice from a terminal, once with `Accept: text/markdown` and once plain, and read what comes back. If both return the same theme HTML, agents are receiving your full page and spending their context budget on your navigation. A readability score on the same page gives you a baseline to measure improvements against.

## Does llms.txt improve AI search visibility?

On its own, the evidence says no. A 300k-domain study in mid-2026 found no measurable citation lift from llms.txt alone, while content negotiation measured roughly 4.2x more effective for accurate retrieval. Publish an llms.txt because it is a useful index of what you have, and treat content negotiation as the mechanism.

## Does AI search optimization only apply to online stores?

No. The measurement in this guide includes a physiotherapy clinic’s services page, which has no price and no product markup: the response went from 13,581 bytes to 1,412, a reduction of 89.6%, with opening hours moving from character offset 10,152 to line 8. Clinics, agencies, docs sites, and public-sector pages all get asked the same kind of factual question.

## How long does AI search optimization take to show results?

The format layer is measurable in days because it changes what the next fetch receives, and answer rate moves in the same week you ship it. The reputational layer, meaning what assistants recall about you without fetching, moves on the timescale of ordinary content and link building. Start with the fast layer, because it is also the one you can prove.

## Will serving agents markdown hurt my Google rankings?

Not when the substance is identical and known search crawlers always receive your canonical HTML. Cloaking means showing search engines different content than users get. Standard HTTP content negotiation delivering the same prices, availability, and policies in a different format is a different thing, and it is the line to stay on.

## Related

- [AI visibility tools: how to choose one](https://rebilder.com/learn/ai-visibility-tools)
- [llms.txt: what it is and whether it works](https://rebilder.com/learn/llms-txt)
- [LLM SEO: what transfers from SEO](https://rebilder.com/learn/llm-seo)
- [What is agentic commerce?](https://rebilder.com/learn/what-is-agentic-commerce)
- [Answer engine optimization tools](https://rebilder.com/learn/answer-engine-optimization-tools)
- [How to get recommended by ChatGPT](https://rebilder.com/learn/get-recommended-by-chatgpt)
- [AI agent user agents in your logs](https://rebilder.com/learn/ai-agent-user-agents)
- [Gateway quickstart](https://rebilder.com/docs/quickstart)
- [Free llms.txt generator](https://rebilder.com/tools/llms-txt-generator)