---
title: "Amazonbot: what Amazon’s crawler does, its user agent and how to control it"
description: Amazonbot improves Amazon services and may train Amazon AI models. Its user agent, robots.txt rules, IP lists, and how Amzn-SearchBot and Amzn-User differ.
canonical_url: https://rebilder.com/learn/amazonbot
last_updated: "2026-11-15"
---

# [Amazonbot: what Amazon’s crawler does, its user agent and how to control it](https://rebilder.com/learn/amazonbot)

> Amazonbot improves Amazon services and may train Amazon AI models. Its user agent, robots.txt rules, IP lists, and how Amzn-SearchBot and Amzn-User differ.

- **Updated:** 2026-11-15
- **Author:** Rebilder
- **Published:** 2026-11-15

**Amazonbot is Amazon’s general web crawler: Amazon says it is used to improve Amazon’s products and services, to provide more accurate information to customers, and that what it collects may be used to train Amazon AI models.** It identifies itself with an `Amazonbot` token, follows `robots.txt`, and crawls from IP addresses Amazon publishes.

Amazon documents it alongside two siblings with different jobs: `Amzn-SearchBot`, which makes your content eligible for search experiences such as Alexa, and `Amzn-User`, which fetches live pages when a customer’s question needs them. Every one of them is controlled by a separate `robots.txt` token and verified against a separate IP list. This guide covers all three, the rules that block training without leaving Alexa, how Amazon reads `robots.txt` and robots meta tags, and how to confirm a request is really Amazon.

## What is Amazonbot?

Amazon’s developer page “Amazon Web Search crawlers” describes how site owners control Amazonbot, Amzn-SearchBot and Amzn-User. On Amazonbot it is brief and direct: the crawler is used to improve Amazon’s products and services, it helps Amazon give customers more accurate information, and its data may be used to train Amazon AI models.

- **It is a training-eligible crawler.** Of the three Amazon agents, Amazonbot is the only one Amazon does not exclude from generative AI training. The page says that both Amzn-SearchBot and Amzn-User do not crawl content for model training.
- **Each setting is independent.** A rule for Amazonbot does not govern Amzn-SearchBot or Amzn-User, and the reverse is true.
- **Changes are not instant.** Amazon says its systems may take about a day to reflect a change to your settings.
- **There is a contact address.** Amazon asks publishers with questions to write to `amazonbot@amazon.com` and to include every relevant domain name in the message.

Amazon’s own documentation and IP lists

```text
Amazon Web Search crawlers:  https://developer.amazon.com/amazonbot
Amazonbot IPs:               https://developer.amazon.com/amazonbot/ip-addresses/
Amzn-SearchBot IPs:          https://developer.amazon.com/amazonbot/searchbot-ip-addresses/
Amzn-User IPs:               https://developer.amazon.com/amazonbot/live-ip-addresses/
```

## Amazonbot vs Amzn-SearchBot vs Amzn-User

The three agents map onto the same split other AI companies use: one crawler with broad uses, one indexer for search answers and one fetcher that acts on a live question. Amazon’s descriptions:

| Agent | What Amazon says it does | Model training | robots.txt |
| --- | --- | --- | --- |
| **Amazonbot** | Improves Amazon products and services and helps provide more accurate information to customers. | May be used to train Amazon AI models. | Follows its own `Amazonbot` group. |
| **Amzn-SearchBot** | Improves search experiences in Amazon products. Allowing it makes your content eligible to appear in search experiences such as Alexa. | Does not crawl for generative AI model training. | Follows its own group. If none names it, it follows the rules you give other search bots. |
| **Amzn-User** | Supports user actions, such as fetching live information from the web to answer an Alexa question on the customer’s behalf. | Does not crawl for generative AI model training. | Because a user can initiate its actions, Amazon says it may not follow all directives. |

The comparison with OpenAI and Anthropic is close but not exact. [GPTBot](/solutions/gptbot) and [ClaudeBot](/learn/claudebot) are described purely as training crawlers, while Amazon describes Amazonbot as improving its products generally, with training as one possible use. If your concern is training, Amazonbot is still the token to block; the indexing and live-answer roles that matter for Alexa visibility belong to the other two.

## Amazonbot user agent strings

All three agents use the same Chrome-shaped string with a different token in the `compatible;` clause. Amazon publishes these examples:

Amazon’s published user-agent examples

```http
# Amazonbot
User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36

# Amzn-SearchBot
User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36

# Amzn-User
User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36
```

`W.X.Y.Z` stands in for a Chrome version that changes, so the only stable parts are the tokens. Note that `Amzn-SearchBot` and `Amzn-User` do not contain the word “Amazonbot”: a log filter or firewall rule written for `amazonbot` misses both of them. Search for all three tokens.

Count each Amazon agent in an access log

```bash
grep -ioE "amazonbot|amzn-searchbot|amzn-user" /var/log/nginx/access.log \
  | tr "[:upper:]" "[:lower:]" | sort | uniq -c | sort -rn
```

## How to block Amazonbot in robots.txt

Amazon says its automated crawling respects the Robots Exclusion Protocol, honoring the user-agent line and `Allow` and `Disallow` directives. The common configurations:

robots.txt: decline training, stay eligible for Alexa search

```text
User-agent: Amazonbot
Disallow: /

User-agent: Amzn-SearchBot
Allow: /
Disallow: /account/
Disallow: /checkout/
```

robots.txt: block every Amazon agent

```text
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
Disallow: /
```

Remember that Amazon reads `robots.txt` per host. A rule on `www.example.com` does nothing for `shop.example.com`, so a store on its own subdomain needs its own file with its own Amazon groups. Three more Amazon behaviors are worth knowing before you rely on the file:

- **Amzn-SearchBot borrows other search bots’ rules.** If your file never names it but allows other search bots, Amazon says it crawls by the rules given to those bots. Name it explicitly if you want a different answer for Amazon.
- **Amzn-User may not follow every directive.** Amazon ties this to its actions being user-initiated. If you need Amzn-User kept off a path, enforce it at your CDN with the token and Amazon’s live-fetch IP list, not only in `robots.txt`.
- **Crawl-delay is ignored.** Amazon states that its agents do not support the directive. To slow Amazonbot down, rate-limit it at your server or CDN.

How Amazon’s agents fetch robots.txt (from Amazon’s page)

```text
- Fetched per host: example.com/robots.txt and site.example.com/robots.txt are read separately.
- A cached copy from the last 30 days may be used instead of a fresh fetch.
- If the file cannot be fetched, the agents behave as if it does not exist.
- Settings changes may take ~24 hours to take effect.
```

> **An unreachable robots.txt means no rules for Amazon** The robots exclusion standard tells crawlers to assume a full disallow when `robots.txt` returns a server error. Amazon describes the opposite default: if its agents cannot fetch the file, they act as if it does not exist. Keep `robots.txt` reliably reachable, including behind any bot protection, or your Amazon rules may not be applied.

## Amazonbot and robots meta tags: noarchive means no training

Amazon’s agents read page-level robots meta tags and link-level `rel=nofollow`. The one to know is `noarchive`, which Amazon defines in its own terms:

| Directive | What Amazon’s agents do |
| --- | --- |
| `noarchive` | Do not use the page for model training. |
| `noindex` | Do not index the page. |
| `none` | Do not index the page. |
| `rel=nofollow` on a link | Respected at the link level. |

Opt one page out of Amazon training while leaving it crawlable

```text
<meta name="robots" content="noarchive">
```

That gives you page-by-page control that `robots.txt` cannot. A publisher can let Amazonbot read a pricing page but mark long-form articles `noarchive`. Bing also reads `noarchive` as an AI-use signal, as the [bingbot guide](/learn/bingbot) explains, so one tag carries the preference to both.

## How to verify Amazonbot with Amazon’s IP lists

Anyone can put `Amazonbot` in a user agent. Amazon publishes a separate list of IP addresses for each agent, and a request is genuine when its source address falls inside the list for the token it claims:

1. Pull the source IP and user agent from the log line or request.
2. Choose the list that matches the token: the Amazonbot list, the Amzn-SearchBot list or the Amzn-User live-fetch list.
3. Check the IP against that list’s prefixes. A match confirms Amazon; no match means someone else is using the name.
4. Reload the lists on a schedule. Prefixes change, so automate the import rather than copying ranges once.

Use the lists to identify Amazon, and `robots.txt` to state your preferences. Blocking Amazon’s addresses also blocks them from reading the file that carries your opt-out. Amazon’s current page documents the IP lists as its verification method; it does not describe a reverse DNS check.

## What blocking Amazonbot changes

- **Block Amazonbot only.** Amazon’s general crawler stops collecting your pages, which removes them from the crawl that may train Amazon AI models. Amzn-SearchBot can still index you for Alexa search, and Amzn-User can still fetch a page for a live question.
- **Block Amzn-SearchBot.** Your content is no longer eligible for Amazon search experiences such as Alexa, which Amazon says depend on allowing it.
- **Block Amzn-User.** Amazon says Amzn-User may not follow every directive, so back the `robots.txt` rule with a CDN block if live fetches for customer questions must stop.
- **Block all three.** Amazon stops crawling, indexing and live-fetching your site, as far as `robots.txt` reaches.

Amazon also connects crawl access to a commercial program. Its crawler page says sites that allow Amazonbot may be eligible for benefits through Amazon Content Partners, aimed at independent content creators. Weigh that if you publish content that earns affiliate revenue.

One more channel points the other way. Amazon is a participant in IndexNow, with its own endpoint listed on indexnow.org, so a URL you announce to Bing through IndexNow is shared with Amazon as well. See [robots.txt sitemap](/learn/robots-txt-sitemap) for how IndexNow fits beside your sitemap.

## How Rebilder handles Amazonbot

Rebilder’s gateway gives AI agents a markdown version of your pages rendered from the sources you connect, at the same URLs your customers use. Amazonbot is on the agent side of that line:

- **Detection.** `@rebilder/agent-detect`, the open-source detector behind the gateway, maps `Amazonbot` to the `amazon` platform as an agent at medium confidence: a user-agent claim is all Amazonbot presents. Its test fixture is transcribed from Amazon’s published string.
- **Serving.** An Amazonbot request for a URL one of your sources answers gets that markdown; a URL with no source falls through to your normal HTML. See [Markdown vs HTML](/help/agent-traffic/markdown-vs-html).
- **Access rules.** A gateway [access rule](/docs/access-control) that names a platform applies only to verified requests, and Amazonbot sends no signature. Write Amazonbot rules with the `unverified` or `*` subjects, and keep `robots.txt` as the training opt-out Amazon reads.
- **The siblings.** `Amzn-SearchBot` and `Amzn-User` are not in the detector’s token list today. They carry browser-shaped user agents, so without an explicit markdown request they classify from their other headers and receive your HTML.

The detector reads headers only. Pair it with the IP-list check at your CDN when you need to know a request truly came from Amazon. [Classification](/docs/classification) sets out every check in order.

## What is Amazonbot?

Amazonbot is Amazon’s web crawler. Amazon says it is used to improve Amazon’s products and services and to provide more accurate information to customers, and that its data may be used to train Amazon AI models. It identifies itself with the `Amazonbot` token and follows `robots.txt`.

## How do I block Amazonbot?

Add `User-agent: Amazonbot` and `Disallow: /` to the `robots.txt` on each host. That does not affect `Amzn-SearchBot` or `Amzn-User`, which have their own tokens. Amazon says changes can take about a day to apply.

## Does Amazonbot train AI models?

Amazon says Amazonbot’s data may be used to train Amazon AI models. It says Amzn-SearchBot and Amzn-User do not crawl content for generative AI model training. To decline training page by page, add the `noarchive` robots meta tag, which Amazon defines as do not use the page for model training.

## Does blocking Amazonbot remove my site from Alexa?

Not by itself. Amazon ties eligibility for search experiences such as Alexa to `Amzn-SearchBot`, and live Alexa answers to `Amzn-User`. Blocking only Amazonbot leaves both of those in place.

## Does Amazonbot respect crawl-delay?

No. Amazon states that its agents do not support the `Crawl-delay` directive. Cap the request rate with a CDN or server rate limit that matches the `Amazonbot` token and Amazon’s IP list.

## How can I tell if Amazonbot traffic is real?

Check the source IP against the list Amazon publishes for that agent at `https://developer.amazon.com/amazonbot/ip-addresses/`. Amzn-SearchBot and Amzn-User have separate lists. A request claiming to be Amazonbot from an address outside its list is not Amazon.

## What is Amzn-SearchBot?

Amazon’s search crawler. Amazon says allowing it makes your content eligible to appear in search experiences such as Alexa, and that it does not crawl content for generative AI model training. If your `robots.txt` does not name it, it follows the rules you set for other search bots.

## What happens if my robots.txt cannot be fetched?

Amazon says its agents then behave as if the file does not exist, which means no restrictions. That differs from the robots exclusion standard’s rule for server errors, so make sure `robots.txt` stays reachable to Amazon’s addresses.

## Related

- [AI agent user agents: the full reference](https://rebilder.com/learn/ai-agent-user-agents)
- [ClaudeBot: Anthropic’s crawler](https://rebilder.com/learn/claudebot)
- [Allow, deny or rate-limit agents](https://rebilder.com/docs/access-control)
- [How agent detection works](https://rebilder.com/docs/classification)
- [Markdown vs HTML for agents](https://rebilder.com/help/agent-traffic/markdown-vs-html)
- [Free AI robots.txt generator](https://rebilder.com/tools/robots-txt-generator)