---
title: "OAI-SearchBot: how OpenAI’s search crawler puts you in ChatGPT search"
description: OAI-SearchBot is the crawler behind ChatGPT search. Its user agent, how to allow it while blocking GPTBot, its IP list, and how to verify the traffic.
canonical_url: https://rebilder.com/learn/oai-searchbot
last_updated: "2026-11-15"
---

# [OAI-SearchBot: how OpenAI’s search crawler puts you in ChatGPT search](https://rebilder.com/learn/oai-searchbot)

> OAI-SearchBot is the crawler behind ChatGPT search. Its user agent, how to allow it while blocking GPTBot, its IP list, and how to verify the traffic.

- **Updated:** 2026-11-15
- **Author:** Rebilder
- **Published:** 2026-11-15

**OAI-SearchBot is OpenAI’s search crawler: it reads public pages so ChatGPT can show and link them when it answers with search results.** OpenAI documents it separately from `GPTBot`, which collects content that may train its models, and from `ChatGPT-User`, which opens a page because a person asked ChatGPT to.

If your goal is to be found in ChatGPT, OAI-SearchBot is the bot that matters. This guide covers the user-agent string OpenAI publishes, a `robots.txt` that welcomes OAI-SearchBot while declining GPTBot, how quickly OpenAI picks up a change, how to check a request against OpenAI’s IP list, and how to keep a CDN bot rule from turning it away. GPTBot itself has [its own guide](/solutions/gptbot).

## What is OAI-SearchBot?

OpenAI’s crawler documentation puts it plainly: “OAI-SearchBot is for search.” It is used “to surface websites in search results in ChatGPT’s search features.” When ChatGPT answers a question with sources, the pages it can cite come from what OAI-SearchBot was allowed to read.

The same entry spells out the consequence of an opt-out. Sites that disallow OAI-SearchBot “will not be shown in ChatGPT search answers, though can still appear as navigational links.” OpenAI recommends allowing OAI-SearchBot in `robots.txt` and “allowing requests from our published IP ranges” if you want to appear.

OpenAI’s own pages, for the current details

```text
Crawler overview:      https://developers.openai.com/api/docs/bots
                       (older links: https://platform.openai.com/docs/bots)
OAI-SearchBot IPs:     https://openai.com/searchbot.json
Publisher FAQ:         https://help.openai.com/en/articles/12627856-publishers-and-developers-faq
```

Two details from that documentation shape the rest of this guide. Each OpenAI setting “is independent of the others”, so a search decision never doubles as a training decision. And when a site allows both OAI-SearchBot and GPTBot, OpenAI may use the results of one crawl for both purposes to avoid fetching the same page twice.

## OAI-SearchBot vs GPTBot vs ChatGPT-User

OpenAI lists four user agents. Three of them read ordinary pages, and each answers a different question about your content:

| User agent | What OpenAI uses it for | robots.txt | IP list | What a block changes |
| --- | --- | --- | --- | --- |
| **OAI-SearchBot** | Surfacing websites in ChatGPT’s search features. | Follows a `User-agent: OAI-SearchBot` group. | `searchbot.json` | Your pages stop appearing in ChatGPT search answers; they can still show as navigational links. |
| **GPTBot** | Crawling content that may be used to train OpenAI’s foundation models. | Follows a `User-agent: GPTBot` group. | `gptbot.json` | Signals that your content should not be used for training. Search is unaffected. |
| **ChatGPT-User** | Visiting a page during a user action in ChatGPT or a Custom GPT. Not automatic crawling. | OpenAI says `robots.txt` rules may not apply, because a user started the action. | `chatgpt-user.json` | Use a firewall rule if you need it out. OpenAI says it does not decide search inclusion. |
| **OAI-AdsBot** | Checking landing pages submitted as ads on ChatGPT. It visits only those pages. | OpenAI’s page names no `robots.txt` control for it; it visits only pages you submit as ads. | `adsbot.json` | Relevant only if you advertise on ChatGPT. |

The row that trips people up is ChatGPT-User. OpenAI states that it “is not used to determine whether content may appear in Search” and points site owners to OAI-SearchBot for search opt-outs. Blocking ChatGPT-User keeps ChatGPT from opening your page on request; it is the wrong lever for search visibility in either direction.

Anthropic draws the same three-way line with `ClaudeBot`, `Claude-SearchBot` and `Claude-User`, described in our [ClaudeBot guide](/learn/claudebot). Perplexity splits indexing from live fetches too; see [PerplexityBot](/learn/perplexitybot).

## OAI-SearchBot user agent string

OpenAI publishes an example string and says the version number may change. Unlike GPTBot’s short string, the current OAI-SearchBot example carries a full desktop Chrome prefix before the `compatible;` clause:

OpenAI’s published OAI-SearchBot examples

```http
# Page fetches
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

# robots.txt fetches add a marker
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot
```

The `robots.txt` marker is there so you can tell a policy check from a page fetch even when your logs omit the path. A burst of marked requests after you edit the file is OpenAI rereading your rules.

Older logs and Rebilder’s detector fixtures also contain a shorter form without the Chrome prefix:

Shorter form, as in Rebilder’s detector fixtures

```http
User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot
```

Because the browser prefix looks like a real desktop visitor, a bot rule that inspects only the start of the string can mistake OAI-SearchBot for a person, and an analytics filter that strips everything after the browser name will lose it. Look for the `OAI-SearchBot` token anywhere in the header, ignoring case and version.

## Allow OAI-SearchBot and block GPTBot in robots.txt

This is the setup OpenAI uses as its own example: appear in ChatGPT search, keep your content out of training. Two groups in the `robots.txt` at your site root express it:

robots.txt: in ChatGPT search, out of training

```text
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /
```

Most sites also have pages no crawler should index, such as carts, account screens and internal search results. Put those exclusions inside the OAI-SearchBot group itself:

robots.txt: search-visible, with private paths excluded

```text
User-agent: *
Disallow: /cart
Disallow: /account/
Disallow: /search

User-agent: OAI-SearchBot
Disallow: /cart
Disallow: /account/
Disallow: /search
Allow: /

User-agent: GPTBot
Disallow: /
```

> **OAI-SearchBot reads one group, not two** Crawlers that follow the robots standard pick the most specific group that names them and skip the rest. Once `User-agent: OAI-SearchBot` exists, the lines under `User-agent: *` stop applying to it. Leave the private paths out of its group and you have opened them to ChatGPT search without meaning to.

The reverse choice is also supported. To stay out of ChatGPT search while allowing training, swap the two directives. To decline everything OpenAI crawls automatically, disallow both tokens. None of these rules governs ChatGPT-User; see the firewall section below for that.

To keep a single page out of ChatGPT search while leaving the rest crawlable, OpenAI’s publisher FAQ points to the `noindex` meta tag, with one condition: OAI-SearchBot has to be allowed to fetch the page to see the tag. A `Disallow` line and a `noindex` tag on the same URL work against each other.

## How long robots.txt changes take for ChatGPT search

OpenAI says it can take about a day from a `robots.txt` update for its search systems to adjust. Plan around that window: a page you open up will not appear in answers the same afternoon, and one you close off may linger briefly.

1. **Edit and deploy** the file at the root of each host. `shop.example.com` and `www.example.com` each serve their own `robots.txt`.
2. **Fetch it from outside** your network with a plain request and confirm a text response, not a login wall, challenge page or redirect loop.
3. **Purge the CDN copy** if your host caches static files, so OpenAI does not keep reading the old rules.
4. **Watch for the marker.** Requests carrying `OAI-SearchBot` plus `robots.txt` in the user agent show that OpenAI has read the new file.
5. **Check ChatGPT a day later** by asking a question your page answers, with search on, and looking for your link among the sources.

Confirm what OAI-SearchBot will read

```bash
curl -sS -o /dev/null -w "%{http_code} %{content_type}\n" https://www.example.com/robots.txt
curl -sS https://www.example.com/robots.txt | sed -n "/OAI-SearchBot/,/^$/p"
```

## OAI-SearchBot IP addresses: verifying a request

The user-agent header is self-declared, so treat it as a name tag rather than an ID. OpenAI publishes a separate JSON file of source ranges for each bot. A request naming OAI-SearchBot should come from `searchbot.json`, not from GPTBot’s or ChatGPT-User’s list:

Read OpenAI’s published OAI-SearchBot ranges

```bash
curl -s https://openai.com/searchbot.json \
  | jq -r ".prefixes[] | .ipv4Prefix // .ipv6Prefix"

# Each prefix entry reads like {"ipv4Prefix": "a.b.c.d/nn"}

# Sibling lists, one per bot:
# https://openai.com/gptbot.json
# https://openai.com/chatgpt-user.json
# https://openai.com/adsbot.json
```

- **Automate the refresh.** The file carries a `creationTime`, and the ranges change. Pull it into your firewall’s IP list on a schedule instead of pasting prefixes once.
- **Match the token to the list.** A request saying OAI-SearchBot from an address outside `searchbot.json` is someone borrowing the name. Rate-limit or block it like any unidentified scraper.
- **Reverse DNS is not part of the method.** OpenAI’s crawler page documents the JSON lists only.
- **No signatures for this bot.** The crawler page does not describe Web Bot Auth for OAI-SearchBot. ChatGPT’s cloud browser, which acts on a user’s behalf, is a different client that OpenAI says CDNs such as Cloudflare and Vercel verify by its request signatures.

## Keep your CDN from blocking OAI-SearchBot

A permissive `robots.txt` means nothing if the edge refuses the request first. Managed bot settings are the usual culprit, and both major platforms now separate search bots from training crawlers:

- **Cloudflare.** Its AI Crawl Control bot reference files OAI-SearchBot under **AI Search** and GPTBot under **AI Crawler**, so you can set **Block** on GPTBot and **Allow** on OAI-SearchBot from the same Crawlers tab. A broader rule elsewhere in the zone, such as a custom rule that challenges unknown automation, can still catch it, so check the security events for the token. Our [Cloudflare AI Crawl Control guide](/learn/cloudflare-ai-crawl-control) walks through the screens.
- **Vercel.** The Firewall’s AI Bots managed ruleset has separate **AI Crawlers** and **AI Assistants** categories. Vercel points to bots.fyi for which bot sits where; it lists OAI-SearchBot as an AI Assistant and GPTBot as an AI Crawler. Denying crawlers while allowing assistants matches the split above.
- **Any WAF.** Pair the token with the IP list in an allow or skip rule, and place it ahead of rules that challenge unknown automation.

Vercel CLI: deny AI crawlers, allow AI assistants

```bash
vercel firewall rules update ai-bots --category ai-crawlers --action deny --yes
vercel firewall rules update ai-bots --category ai-assistants --action allow --yes
vercel firewall publish
```

Cloudflare custom rule (Skip): genuine OAI-SearchBot only

```text
(http.user_agent contains "OAI-SearchBot" and ip.src in $openai_searchbot)

# $openai_searchbot is an account IP list loaded from searchbot.json
```

ChatGPT-User is the case where the edge is your only control. If you want it kept out, match the `ChatGPT-User` token with `chatgpt-user.json` and block there, since OpenAI says `robots.txt` may not apply to it.

## Measure what ChatGPT search sends you

Two different things are worth counting. OAI-SearchBot requests tell you what OpenAI is reading. Clicks from ChatGPT answers tell you what it is sending back. OpenAI’s publisher FAQ says ChatGPT adds `utm_source=chatgpt.com` to referral URLs, so those visits are visible in ordinary analytics.

Crawls and click-throughs in an nginx access log

```bash
# Pages OAI-SearchBot read most (excluding robots.txt checks)
grep "OAI-SearchBot" /var/log/nginx/access.log | grep -v "robots.txt;" \
  | cut -d\" -f2 | cut -d" " -f2 | sort | uniq -c | sort -nr | head

# Landing pages reached from ChatGPT answers
grep "utm_source=chatgpt.com" /var/log/nginx/access.log \
  | cut -d\" -f2 | cut -d" " -f2 | cut -d"?" -f1 | sort | uniq -c | sort -nr | head
```

Compare the two lists. A page OAI-SearchBot reads often but nobody clicks through to may be cited without being persuasive. A page that gets clicks but rarely appears in the crawl list is worth linking more prominently from pages OpenAI already reads.

## Serve OAI-SearchBot your real facts with Rebilder

Being crawled is the start. What ChatGPT says about your prices, stock and policies depends on what OAI-SearchBot finds in the response. Rebilder’s gateway answers it with markdown built from your own product, service and policy sources, so the facts arrive as text instead of being buried in layout and scripts.

- **Recognition.** `@rebilder/agent-detect` matches the `oai-searchbot` token and assigns it to the `chatgpt` platform at medium confidence, alongside `GPTBot` and `ChatGPT-User`. [Classification](/docs/classification) lists every check the detector runs.
- **Serving.** When a configured source covers the URL, OAI-SearchBot gets the markdown rendering; any other URL passes through to your HTML untouched. Googlebot and other search crawlers always receive canonical HTML.
- **Reporting.** Once events reach the Console, these requests appear under ChatGPT, each marked with whether it got markdown or HTML. It groups all three OpenAI bots under that one platform, so use the log commands above when you need them separated.
- **Access rules.** Be careful with a blanket `unverified` deny in the [access policy](/docs/access-control). OAI-SearchBot sends no signature, so that rule turns it away along with every other unsigned agent, which is the opposite of what a site seeking ChatGPT visibility wants.
- **Readability check.** The free [scan](/scan) reads your `robots.txt` and reports an opt-out when the file blocks AI assistants by name; OAI-SearchBot is one of the assistant tokens it checks. Declining training crawlers such as GPTBot does not change the score.

## What is OAI-SearchBot?

OAI-SearchBot is the crawler OpenAI uses to surface websites in ChatGPT’s search features. If it can read your pages, ChatGPT can show and link them in search answers. It follows `robots.txt` and sends requests from the ranges in `https://openai.com/searchbot.json`.

## Is OAI-SearchBot the same as GPTBot?

No. GPTBot crawls content that may be used to train OpenAI’s foundation models. OAI-SearchBot crawls for ChatGPT search. Each has its own `robots.txt` token and its own IP list, and OpenAI states that each setting is independent of the other.

## Does blocking GPTBot remove my site from ChatGPT search?

No. Search inclusion depends on OAI-SearchBot. A `robots.txt` that disallows GPTBot and allows OAI-SearchBot keeps your content out of training while leaving it eligible for ChatGPT search answers.

## How do I allow OAI-SearchBot but block GPTBot?

Add `User-agent: OAI-SearchBot` with `Allow: /`, then `User-agent: GPTBot` with `Disallow: /`. If your wildcard group excludes private paths, repeat those `Disallow` lines inside the OAI-SearchBot group, because a named group replaces the wildcard for that bot.

## How long does a robots.txt change take to affect ChatGPT search?

OpenAI says it can take about a day after a `robots.txt` update for its systems to adjust search results. Requests from OAI-SearchBot carrying a `robots.txt` marker in the user agent show when it has reread the file.

## Does ChatGPT-User follow robots.txt?

Not reliably. OpenAI says that because ChatGPT-User acts on a user’s request, `robots.txt` rules may not apply. It also says ChatGPT-User does not determine search inclusion. To block it, match its token and `https://openai.com/chatgpt-user.json` at your firewall.

## Can my site appear in ChatGPT search if OAI-SearchBot is blocked?

Only as a navigational link. OpenAI says opted-out sites will not be shown in ChatGPT search answers, though they can still appear as navigational links.

## Does Rebilder detect OAI-SearchBot?

Yes. The open-source detector matches the `OAI-SearchBot` token and attributes it to the ChatGPT platform at medium confidence. The gateway serves it markdown wherever one of your sources covers the URL.

## Related

- [AI agent user agents: the full reference](https://rebilder.com/learn/ai-agent-user-agents)
- [How to get recommended by ChatGPT](https://rebilder.com/learn/get-recommended-by-chatgpt)
- [ClaudeBot and Anthropic’s three bots](https://rebilder.com/learn/claudebot)
- [Track AI traffic to your site](https://rebilder.com/learn/track-ai-traffic)
- [How agent detection works](https://rebilder.com/docs/classification)
- [Which agents Rebilder detects](https://rebilder.com/help/agent-traffic/which-agents-detected)