---
title: "meta-externalagent: Meta’s AI crawler and the four bots beside it"
description: meta-externalagent is Meta’s crawler for AI training and product indexing. Its user agent, robots.txt rules, and how it differs from facebookexternalhit.
canonical_url: https://rebilder.com/learn/meta-externalagent
last_updated: "2026-11-15"
---

# [meta-externalagent: Meta’s AI crawler and the four bots beside it](https://rebilder.com/learn/meta-externalagent)

> meta-externalagent is Meta’s crawler for AI training and product indexing. Its user agent, robots.txt rules, and how it differs from facebookexternalhit.

- **Updated:** 2026-11-15
- **Author:** Rebilder
- **Published:** 2026-11-15

**meta-externalagent is the crawler Meta uses for “training foundation AI models or improving products by indexing content directly.”** It sends the lowercase `meta-externalagent` token in its user agent, it reads `robots.txt`, and it is only one of five crawlers Meta lists for site owners.

The five do very different jobs. One builds the link previews people see when they share your URL on Facebook, Instagram or Messenger. One indexes pages so Meta AI can cite them. One fetches a page because a user asked, and may not consult `robots.txt` at all. This guide explains each, gives the `robots.txt` rules to decline AI training without breaking your share previews, and shows what each block does to your presence in Meta AI’s answers.

## What is meta-externalagent?

Meta describes its crawlers on a single webmasters page. The entry for this one says the Meta-ExternalAgent crawler “crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.” Two uses sit in that sentence: collecting material for model training, and indexing pages for Meta’s own products.

Meta also states its preferred control channel. It aims to make preferences easy to express “by using industry-standard practices like robots.txt rather than non-standard formats like NoAI tags.” A `noai` meta tag is therefore not the way to reach this crawler; a `robots.txt` group is.

Meta’s documentation and contact

```text
Meta web crawlers:   https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers
Sharing Debugger:    https://developers.facebook.com/tools/debug
Questions:           webmasters@meta.com
```

## meta-externalagent vs meta-externalfetcher vs facebookexternalhit

Here are all five crawlers on Meta’s page, with Meta’s own description of each and what happens when you turn it away:

| Crawler token | What Meta says it does | robots.txt | Turning it away means |
| --- | --- | --- | --- |
| `facebookexternalhit` | Reads a page shared in Meta’s apps and gathers, caches and displays its title, description and thumbnail. | Followed for ordinary fetches; Meta says it might bypass the file for security or integrity checks such as malware scanning. | Shared links lose their preview card, or show a bare URL. |
| `meta-webindexer` | Navigates the web to improve the quality of Meta AI search results. | Follows its own group. | Meta AI has less reason to cite and link your pages. Meta says allowing it helps it do exactly that. |
| `meta-externalads` | Crawls for use cases such as improving advertising and other business-related products. | Follows its own group; it appears in Meta’s example file. | Opts your pages out of those advertising and business uses. |
| `meta-externalagent` | Training foundation AI models, or improving products by indexing content directly. | Follows its own group; it appears in Meta’s example file. | Your future pages are declined for Meta’s model training and this product indexing. |
| `meta-externalfetcher` | Fetches individual links at a user’s request, including helping AI navigate websites to complete tasks. | Meta says it may bypass `robots.txt` because a user requested the fetch. | Only an edge block reliably stops it; then Meta AI cannot open your page when a user asks. |

The practical split mirrors other AI platforms. A training crawler, a search indexer and a user-triggered fetcher each get their own token, so you can answer each question separately. Compare OpenAI’s version in [OAI-SearchBot](/learn/oai-searchbot) and Anthropic’s in [ClaudeBot](/learn/claudebot). What Meta adds is the link-preview bot, which predates its AI products and matters for every site whose URLs get shared.

## meta-externalagent user agent string

Meta prints two forms for each crawler: one with a reference to its documentation, one bare. Note that the reference is a relative path, not a full URL:

Meta’s published user agents

```http
User-Agent: meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers)
User-Agent: meta-externalagent/1.1

User-Agent: meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers)
User-Agent: meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers)
User-Agent: meta-externalads/1.1 (+/documentation/sharing/webmasters/web-crawlers)

User-Agent: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
User-Agent: facebookexternalhit/1.1
```

- **No browser prefix.** These strings do not begin with `Mozilla/5.0`, so a filter that looks only for browser-shaped headers will not see them, and a rule that challenges non-browser clients will catch all five.
- **Lowercase tokens.** Meta writes the tokens in lowercase in the header and in title case in its headings. Match without regard to case.
- **Watch the prefix overlap.** `meta-externalagent` and `meta-externalfetcher` share the start `meta-external`, as does `meta-externalads`. A rule on that shared prefix catches all three; write the full token when you mean one.

## How to block meta-externalagent in robots.txt

Meta says to block a crawler by adding a disallow for it in `robots.txt`. For most businesses the useful configuration declines training while keeping share previews and Meta AI citations:

robots.txt: no training crawl, previews and Meta AI citations kept

```text
User-agent: meta-externalagent
Disallow: /

User-agent: meta-webindexer
Allow: /

User-agent: facebookexternalhit
Allow: /
```

Meta’s own example takes a lighter approach: allow the crawler everywhere except one directory. That suits a site that is happy to be indexed but wants a members area or draft content kept back:

robots.txt: the shape of Meta’s published example

```text
User-agent: meta-externalagent
Allow: /
Disallow: /private/

User-agent: meta-externalads
Allow: /
Disallow: /private/
```

To keep every Meta AI crawler out while leaving link previews alone, list the AI tokens in one group. Leave `facebookexternalhit` out of it:

robots.txt: decline Meta’s AI crawlers, keep previews

```text
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: meta-externalads
User-agent: meta-externalfetcher
Disallow: /
```

> **Allow up to a day** Meta says its crawlers may cache `robots.txt` for up to a day, so a change can take that long to apply. The `meta-externalfetcher` line above states your preference, but Meta says that crawler may not read it; the next section covers the edge rule.

Each named group stands alone. If your general `User-agent: *` group keeps crawlers out of checkout or account pages, copy those lines into any Meta group that says `Allow: /`, or the Meta crawler will treat them as open.

## meta-externalfetcher and robots.txt

Meta’s wording for the fetcher is direct: it “fetches individual links at a user’s request and supports product functions such as evaluating and improving agentic AI capabilities”, including “helping AI navigate websites to complete tasks for users. Accordingly, this crawler may bypass robots.txt rules.”

Two consequences follow. First, a `robots.txt` group for `meta-externalfetcher` is a request, not a control. Second, the fetcher is the bot that acts while someone is waiting for an answer, so blocking it is the Meta decision most likely to cost you a customer interaction. If you still want it out:

1. **Cloudflare.** The AI Crawl Control bot reference lists Meta-ExternalFetcher in the **AI Assistant** category and Meta-ExternalAgent as an **AI Crawler**, so each has its own allow or block action on the Crawlers tab.
2. **Vercel.** Its bot catalog, bots.fyi, files the fetcher under **User Initiated** rather than AI Crawler, so check which AI Bots category covers it before relying on the managed ruleset. A custom rule on the `meta-externalfetcher` user agent is unambiguous.
3. **Your own server.** A rule on the full token, returning a refusal, works anywhere. Keep `/robots.txt` itself reachable.

Cloudflare WAF expression: Meta’s fetcher only

```text
lower(http.user_agent) contains "meta-externalfetcher"
```

## Keep link previews working: facebookexternalhit

When someone pastes your URL into a post or a chat, `facebookexternalhit` fetches the page and builds the preview card from it. Blocking AI crawlers should never touch it, and the common way it breaks is a broad bot rule rather than a deliberate choice. Meta’s requirements for a working preview:

- **Compression.** Your server must support gzip and deflate encodings.
- **Tags near the top.** Open Graph properties must appear within the first megabyte of the response, or they are cut off.
- **Speed.** The crawler must be able to fetch the page within a few seconds, or the preview fails.
- **Range requests.** Either honor the byte range the crawler asks for with all required properties inside it, or ignore the `Range` header and return the whole page.
- **Allow-listing.** Meta suggests allowing the crawler by user agent or by IP address, and calls the IP approach more secure.

Meta’s command to simulate a preview fetch

```bash
curl -v --compressed -H "Range: bytes=0-524288" -H "Connection: close" \
  -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" "$URL"
```

If a preview is stale or empty after you fix the page, run the URL through Meta’s Sharing Debugger to force a fresh fetch. On Vercel, bots.fyi files `facebookexternalhit` under **Preview**, outside the AI categories, so denying AI crawlers there leaves previews intact. A rule that blocks every non-browser user agent does not.

## What blocking each Meta bot does to Meta AI answers

- **Block meta-externalagent.** You decline Meta’s model training and the product indexing this crawler performs. Meta’s page attributes Meta AI citations to a different crawler, so this block alone is mainly a training decision.
- **Block meta-webindexer.** Meta says allowing it “helps us cite and link to your content in Meta AI’s responses”, so a block makes your pages less likely to appear as sources there.
- **Block meta-externalfetcher at the edge.** When a user asks Meta AI about your page or to complete a task on your site, the fetch fails and the answer goes on without you.
- **Block facebookexternalhit.** Meta AI is unaffected, but every share of your link on Facebook, Instagram and Messenger loses its preview.

A store, restaurant or service business usually wants the opposite of a blanket block: previews on, citations on, the fetcher allowed, and a training decision made on its own merits. A publisher selling its text may decline both meta-externalagent and meta-webindexer.

## How to verify Meta crawler traffic

Meta’s current crawler page recommends IP-based allow-listing as the more secure option but does not print an address list itself. Earlier versions of the page pointed to the address ranges announced by Meta’s own network for this check, so a common test is whether the source address belongs to the autonomous system Meta operates:

Does a source IP belong to Meta’s network?

```bash
# Meta announces its address space as AS32934
ip=203.0.113.7
whois -h whois.cymru.com " -v $ip" | tail -n1
# Genuine Meta crawler traffic reports AS 32934 in the first column
```

- **Combine token and network.** A `meta-externalagent` header from a hosting provider’s range is not Meta.
- **Let your CDN do it where it can.** Cloudflare assigns its own detection IDs to Meta-ExternalAgent and Meta-ExternalFetcher, and bots.fyi marks Meta’s AI crawler as verified.
- **Nothing signed.** Meta’s page does not describe Web Bot Auth signatures for any of these crawlers.

## How Rebilder handles meta-externalagent

Rebilder’s gateway gives Meta’s AI bots a clean, text-first version of your pages, built from your product, service and policy sources, so what Meta indexes and what Meta AI reads on a user’s behalf matches your own records. What the detector recognizes today:

| Meta token | Rebilder classification | What the gateway serves |
| --- | --- | --- |
| `meta-externalagent` | AI agent, `meta` platform, medium confidence | Markdown wherever a source covers the URL; HTML elsewhere |
| `meta-externalfetcher` | AI agent, `meta` platform, medium confidence | Markdown wherever a source covers the URL; HTML elsewhere |
| `meta-webindexer` | AI agent, `meta` platform, medium confidence | Markdown wherever a source covers the URL; HTML elsewhere |
| `meta-externalads` | Crawler, `meta-ads` platform | Your normal HTML, always |
| `facebookexternalhit` | Crawler, `meta-preview` platform | Your normal HTML with your Open Graph tags, always |

- **Source of the patterns.** The five Meta entries in `@rebilder/agent-detect` and its test fixtures are transcribed from Meta’s crawler page. See [Classification](/docs/classification).
- **Why two are crawlers.** Link previews are built from your page’s own title, description and image, and an ad crawler should see the page your customers see. Both always get your HTML, like Googlebot and Microsoft’s preview fetcher.
- **In the Console**, the three AI bots appear together under Meta AI. Link previews and the ad crawler are counted as crawlers on their own rows, so they never inflate the Meta AI figure.
- **Access rules.** None of the Meta AI bots sends a signature, so a `meta` platform rule in the [access policy](/docs/access-control) will not match them; use the `unverified` subject or `*` if you want to rate-limit them at the gateway. Crawlers are never subject to the access policy, so a rule for every agent cannot break your share previews.
- **Readability scan.** The [free scan](/scan) treats meta-externalagent as a training crawler, so a block on it does not move your result. It counts meta-externalfetcher among the assistant tokens it checks.

## What is meta-externalagent?

meta-externalagent is a Meta web crawler. Meta says it crawls for use cases such as training foundation AI models or improving products by indexing content directly. It identifies itself with the `meta-externalagent` token and follows `robots.txt`.

## Is meta-externalagent the same as facebookexternalhit?

No. `facebookexternalhit` builds the preview card when a link is shared on Facebook, Instagram or Messenger. `meta-externalagent` is an AI training and indexing crawler. They have separate tokens, so you can block one and allow the other.

## Will blocking meta-externalagent break Facebook link previews?

No. Previews come from `facebookexternalhit`. Keep that token allowed in `robots.txt` and in your CDN bot rules, and previews keep working while meta-externalagent is declined.

## Does meta-externalfetcher respect robots.txt?

Not necessarily. Meta says it may bypass `robots.txt` rules because it fetches links at a user’s request. Use a CDN or server rule on the `meta-externalfetcher` user agent if you need it blocked.

## How do I block meta-externalagent?

Add `User-agent: meta-externalagent` and `Disallow: /` to your `robots.txt`. Meta notes its crawlers may hold a cached copy of the file for up to a day. To stay citable in Meta AI, leave `meta-webindexer` allowed.

## What is Meta-WebIndexer?

A Meta crawler that navigates the web to improve Meta AI search results. Meta says allowing it in `robots.txt` helps Meta cite and link to your content in Meta AI’s responses.

## Does Rebilder detect meta-externalagent?

Yes. The detector matches `meta-externalagent`, `meta-externalfetcher` and `meta-webindexer` to the Meta platform at medium confidence, and the gateway can serve them markdown. It classifies `facebookexternalhit` and `meta-externalads` as crawlers, so they always receive your normal HTML.

## Related

- [AI agent user agents: the full reference](https://rebilder.com/learn/ai-agent-user-agents)
- [OAI-SearchBot and ChatGPT search](https://rebilder.com/learn/oai-searchbot)
- [Bytespider: ByteDance’s crawler](https://rebilder.com/learn/bytespider)
- [How agent detection works](https://rebilder.com/docs/classification)
- [Allow, deny or rate-limit agents](https://rebilder.com/docs/access-control)
- [Which agents Rebilder detects](https://rebilder.com/help/agent-traffic/which-agents-detected)