---
title: "Applebot: Apple’s web crawler, Applebot-Extended and how to control both"
description: Applebot crawls for Spotlight, Siri and Safari. Its user agent, the Applebot-Extended training opt-out, IP and reverse DNS checks, and robots.txt rules.
canonical_url: https://rebilder.com/learn/applebot
last_updated: "2026-11-15"
---

# [Applebot: Apple’s web crawler, Applebot-Extended and how to control both](https://rebilder.com/learn/applebot)

> Applebot crawls for Spotlight, Siri and Safari. Its user agent, the Applebot-Extended training opt-out, IP and reverse DNS checks, and robots.txt rules.

- **Updated:** 2026-11-15
- **Author:** Rebilder
- **Published:** 2026-11-15

**Applebot is Apple’s web crawler: the pages it collects power the search built into Spotlight, Siri and Safari, and Apple says the same data may also help train the foundation models behind Apple Intelligence.** It identifies itself with an `Applebot` token inside a Safari-style user agent, follows `robots.txt`, and comes from Apple addresses you can confirm by reverse DNS.

Apple separates the two uses with a second token, `Applebot-Extended`, which never visits your site at all. You set it in `robots.txt` to decide whether Applebot’s crawl may be used for model training, without leaving Apple’s search results. This guide covers both tokens, the exact rules, the Googlebot fallback that catches many sites out, how to verify Applebot traffic, and what each control changes in Apple’s products.

## What is Applebot?

Apple’s support article “About Applebot” describes three uses for what Applebot crawls. Knowing which one you care about tells you which control to reach for:

1. **Search across Apple devices.** Crawled data powers search in Spotlight, Siri and Safari. Apple says allowing Applebot lets your content appear in search results for Apple users around the world.
2. **Training Apple’s foundation models.** The data may be used to help train the models behind generative features across Apple products, including Apple Intelligence, Services and Developer Tools. `Applebot-Extended` is the opt-out.
3. **Context for AI-generated answers.** Apple may use crawled content as up-to-date context when its models generate output, for example broad world-knowledge answers in Siri and Search that link to the sources used. The `nosnippet` robots meta tag is the opt-out for specific content.

Apple’s own documentation, for the live details

```text
About Applebot:      https://support.apple.com/en-us/119829
Applebot IP ranges:  https://search.developer.apple.com/applebot.json
Contact:             applebot@apple.com
```

For a store or service business, the first use is usually the one worth protecting. When someone asks Siri about a product or searches in Spotlight, Applebot’s crawl is what Apple’s search draws on.

## Applebot vs Applebot-Extended

The two names look like two crawlers. They are one crawler and one permission. Apple states that Applebot-Extended does not crawl webpages; it is only used to decide how data that Applebot already crawled may be used.

| Control | Visits your site? | What it decides | Effect on Apple search |
| --- | --- | --- | --- |
| `User-agent: Applebot` group | Yes, this is the crawler. | Which pages Applebot may fetch at all. | Disallowed pages cannot be crawled for Spotlight, Siri and Safari. |
| `User-agent: Applebot-Extended` group | No. Apple says it does not crawl. | Whether Applebot’s crawl may train Apple’s generative foundation models. | None. Apple says pages that disallow it can still appear in search, and its rules are not considered in ranking. |
| `nosnippet` robots meta tag | Read by Applebot on the page. | No description or web answer for the page, and no use as context for AI-generated output. | The page can still be suggested, with its title only. |
| `isAccessibleForFree: false` in JSON-LD | Read by Applebot on the page. | Marks paywalled content, which Apple will not use as context for AI-generated output. | The page stays eligible for search results. |

Compare Google, which runs the same pattern with `Google-Extended`: a token you set in `robots.txt` that governs use of a crawl rather than the crawl itself. The [AI agent user agents guide](/learn/ai-agent-user-agents) lines up every major vendor’s tokens side by side.

## Applebot user agent strings

Applebot presents itself as Safari with an Applebot clause appended. Apple publishes a general format and examples for desktop and mobile:

Applebot user agents (Apple’s published examples)

```http
# General format
Mozilla/5.0 (Device; OS_version) AppleWebKit/WebKit_version (KHTML, like Gecko) Version/Safari_version [Mobile/Mobile_version] Safari/WebKit_version (Applebot/Applebot_version; +http://www.apple.com/go/applebot)

# Desktop
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)

# Mobile
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 17_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Mobile/15E148 Safari/604.1 (Applebot/0.1; +http://www.apple.com/go/applebot)
```

- **Look for the token, not the browser.** Everything before the Applebot clause is an ordinary Safari string, and Apple notes it occasionally updates the browser version it advertises. A rule pinned to the `Version/` number breaks; a rule matching `Applebot` does not.
- **Do not filter on “Safari”.** Bot-blocking rules that treat every Safari user agent as a person will count Applebot as a visitor and skew your analytics.
- **Watch for `iTMS`.** Apple also sends a special crawler with the user agent `iTMS` from `applebot.apple.com` hosts. Apple says it does not follow `robots.txt` because it only crawls URLs associated with content registered on Apple Podcasts, such as a podcast feed you submitted.

You will never see `Applebot-Extended` in a log. Because it does not crawl, it never arrives as a user agent; it exists only as a group name in `robots.txt`.

## Applebot robots.txt rules

Apple says Applebot respects standard `robots.txt` directives targeted at it in general search crawls. Pick the configuration that matches what you want from Apple:

robots.txt: stay in Apple search, opt out of model training

```text
User-agent: Applebot
Allow: /
Disallow: /account/
Disallow: /checkout/

User-agent: Applebot-Extended
Disallow: /
```

That is the configuration most businesses want: Spotlight, Siri and Safari can still find the product, service and policy pages, and nothing is offered for training. Applebot-Extended also accepts path rules, so you can withhold one section from training while leaving the rest available:

robots.txt: withhold one directory from training only

```text
User-agent: Applebot-Extended
Disallow: /guides/
```

robots.txt: leave Apple entirely

```text
User-agent: Applebot
Disallow: /

User-agent: Applebot-Extended
Disallow: /
```

Blocking Applebot alone already stops it fetching your pages, so there is nothing new for training either. The second group is belt and braces: it records your training choice explicitly in case you later reopen Applebot for search.

## The Googlebot fallback: when your Google rules apply to Apple

Apple documents one behavior most crawlers do not have. If your `robots.txt` does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot instructions. In that case it uses the Googlebot group, not your `User-agent: *` group.

robots.txt: Applebot reads the Googlebot group here

```text
User-agent: Googlebot
Disallow: /internal-search/
Disallow: /lp/google-ads/

User-agent: *
Disallow: /
```

In that file Applebot is allowed almost everywhere, because it adopts the Googlebot group, even though the site owner probably meant `*` to close the door on everyone but Google. The reverse surprise is just as common: a strict Googlebot group written for a Google-specific problem quietly restricts Apple too.

> **Name Applebot whenever you name Googlebot** If your file has a Googlebot group, add an explicit `User-agent: Applebot` group with the rules you want Apple to follow. Then the fallback never applies, and a later edit for Google cannot change what Apple does.

## What blocking Applebot or Applebot-Extended changes

- **Disallow Applebot-Extended only.** Apple will not use your content to train its foundation models. Your pages remain discoverable through Spotlight, Siri and Safari, and Apple says the choice is not considered in ranking.
- **Add `nosnippet` to specific pages.** Apple shows only the title for those pages and will not use them as context when its models generate answers. Use it on content you want found but not summarized.
- **Mark paywalled pages.** `isAccessibleForFree: false` in page-level JSON-LD keeps the page eligible for results while excluding it from AI context. Apple says section-level markup with `hasPart` is not supported, so the signal covers the whole page.
- **Disallow Applebot.** Apple can no longer crawl the pages, so they drop out of what Spotlight, Siri and Safari can show. For a business that customers find through an iPhone, that is a real loss.
- **Use `noindex` for individual pages.** Applebot will not index the page, and it will not appear in Spotlight or Siri Suggestions. Unlike a `robots.txt` block, Applebot can still read the page to see the tag.

Page-level Applebot controls

```text
<!-- In the <head>: applies to every crawler that reads robots meta -->
<meta name="robots" content="nosnippet">

<!-- Applebot only -->
<meta name="applebot" content="nosnippet">

# As an HTTP header, for PDFs and other non-HTML files
X-Robots-Tag: applebot: nosnippet
```

## How to verify Applebot: reverse DNS and Apple’s IP ranges

Apple gives two ways to confirm a request that claims to be Applebot. Use either; for a firewall rule, the IP list is easier to automate.

Reverse DNS, per Apple’s example

```bash
$ host 17.58.101.179
179.101.58.17.in-addr.arpa domain name pointer 17-58-101-179.applebot.apple.com.

$ host 17-58-101-179.applebot.apple.com
17-58-101-179.applebot.apple.com has address 17.58.101.179
```

- **Reverse DNS.** Genuine Applebot traffic generally resolves to a host in the `applebot.apple.com` domain. Confirm with a forward lookup that the name returns the original address.
- **Apple’s CIDR list.** Match the source IP against the prefixes in Apple’s published JSON file. Refresh your copy automatically, because Apple can change the prefixes.
- **The iTMS crawler uses the same domain.** A request from an `applebot.apple.com` host with the `iTMS` user agent is Apple’s Podcasts crawler, not a spoofed Applebot.

Fetch Apple’s published Applebot ranges

```bash
curl -s https://search.developer.apple.com/applebot.json
```

## Applebot crawl rate and JavaScript rendering

Applebot does not follow `Crawl-delay`. Apple says its crawl rate adjusts automatically when a site slows down or returns errors, and that it caches crawled content to avoid fetching what has not changed. If you need a hard ceiling, set a rate limit at your CDN keyed on the `Applebot` token and verified Apple addresses.

Applebot may render your pages in a browser. Apple warns that if JavaScript, CSS, XHR requests or other resources a page needs are blocked in `robots.txt`, it may not render the content properly. Two ways to stay safe:

- **Leave rendering resources open.** Do not disallow your script, style or API paths for Applebot if the page needs them to show prices, stock or descriptions.
- **Degrade gracefully.** Put the facts that matter in the HTML the server sends, so the page still makes sense if a resource fails to load. That serves every non-rendering client, not just Apple.

See the server HTML Applebot receives before rendering

```bash
curl -s -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)" \
  https://your-site.example/products/example | grep -i -E "price|in stock"
```

## How Rebilder handles Applebot

Apple describes Applebot as a search crawler first, and Rebilder treats it as one. The open-source detector, `@rebilder/agent-detect`, lists `Applebot` among its crawler tokens alongside Googlebot and bingbot:

- **Canonical HTML, always.** A request carrying the `Applebot` token classifies as a crawler on the `applebot` platform, and the gateway passes it to your normal HTML; a markdown `Accept` header does not change that. Apple indexes the same page your customers see.
- **No pattern for Applebot-Extended.** The token never arrives as a user agent, so the detector has nothing to match. Your training choice lives in `robots.txt`, where Apple reads it.
- **A ready-made rule.** The free [robots.txt generator](/tools/robots-txt-generator) lists `Applebot-Extended` as a training control that does not affect Apple search results, so you can opt out of training without touching Applebot.
- **Visible in the Console.** Applebot requests appear in **AI visits** as **Crawler** rows served the **Regular page**. The help article [Which agents Rebilder detects](/help/agent-traffic/which-agents-detected) lists the other crawlers treated the same way.

The gateway’s markdown path is for AI agents that read on a person’s behalf; search crawlers such as Applebot stay on HTML by design. The detector checks headers, not source addresses, so pair it with the reverse DNS or IP check above when identity matters. [Classification](/docs/classification) documents how the detector orders its checks.

## What is Applebot?

Applebot is Apple’s web crawler. The data it collects powers search in Spotlight, Siri and Safari, and Apple says it may also help train Apple’s foundation models. It identifies itself with an `Applebot` token in a Safari-style user agent and follows `robots.txt`.

## What is Applebot-Extended?

A `robots.txt` token, not a crawler. Disallowing `Applebot-Extended` tells Apple not to use content Applebot crawled to train its generative foundation models. It does not crawl pages, and Apple says it does not affect whether your pages appear in search.

## Does blocking Applebot-Extended remove my site from Siri and Spotlight?

No. Apple states that pages disallowing Applebot-Extended can still be included in search results, and that its rules are not considered in ranking. Only disallowing `Applebot` itself stops Apple crawling your pages for Spotlight, Siri and Safari.

## How do I verify that a request is really Applebot?

Run a reverse DNS lookup on the source IP: genuine traffic generally resolves to a host in `applebot.apple.com`. Confirm with a forward lookup, or match the IP against Apple’s published list at `https://search.developer.apple.com/applebot.json`.

## Does Applebot follow my Googlebot rules?

Yes, if your `robots.txt` has no Applebot group. Apple says that when the file mentions Googlebot but not Applebot, Applebot follows the Googlebot instructions. Add an explicit `User-agent: Applebot` group to control Apple separately.

## Does Applebot support crawl-delay?

No. Apple says Applebot does not follow `Crawl-delay` and instead adjusts its crawl rate automatically when a site slows down or returns errors. Use a rate limit at your CDN if you need a hard ceiling.

## Does Applebot run JavaScript?

It may. Apple says Applebot can render pages in a browser, and that blocking the JavaScript, CSS or XHR a page needs can stop it rendering properly. Keep those resources crawlable, and keep key facts in the server HTML.

## Related

- [AI agent user agents: the full reference](https://rebilder.com/learn/ai-agent-user-agents)
- [Bingbot: user agents and verification](https://rebilder.com/learn/bingbot)
- [Free AI robots.txt generator](https://rebilder.com/tools/robots-txt-generator)
- [How agent detection works](https://rebilder.com/docs/classification)
- [Which agents Rebilder detects](https://rebilder.com/help/agent-traffic/which-agents-detected)
- [Gateway quickstart](https://rebilder.com/docs/quickstart)