---
title: "WordPress robots.txt: where it lives, how to edit it and what to block"
description: Where WordPress robots.txt lives, the default rules core writes, how to edit it with SFTP, Yoast, Rank Math or AIOSEO, and how to add AI crawler rules.
canonical_url: https://rebilder.com/learn/wordpress-robots-txt
last_updated: "2026-11-15"
---

# [WordPress robots.txt: where it lives, how to edit it and what to block](https://rebilder.com/learn/wordpress-robots-txt)

> Where WordPress robots.txt lives, the default rules core writes, how to edit it with SFTP, Yoast, Rank Math or AIOSEO, and how to add AI crawler rules.

- **Updated:** 2026-11-15
- **Author:** Rebilder
- **Published:** 2026-11-15

**WordPress robots.txt is the plain-text file at `/robots.txt` that tells crawlers which parts of a WordPress site they may fetch, and on most sites it does not exist on disk at all.** When no physical file is present, WordPress generates one on every request, and core, your SEO plugin and your CDN can each add lines to it.

This guide covers the default rules, how to view and edit the file, what never to block, rules for AI crawlers such as GPTBot and ClaudeBot, and the multisite and caching cases that make a correct file look wrong in production.

## How WordPress robots.txt works: the virtual file

WordPress registers a rewrite rule that maps `/robots.txt` to its own front controller. When a request arrives and no real file by that name sits in the web root, the server hands the request to WordPress, the template loader sees a robots query and fires the `do_robots` action, and core’s `do_robots()` function prints the response as `text/plain`.

According to WordPress’s developer reference, `do_robots()` fires the `do_robotstxt` action, builds a short default, then passes it through the `robots_txt` filter with a flag saying whether the site is public, and prints whatever the filters return. Every plugin that “edits” a virtual robots.txt is hooking that filter or that action.

> **A physical file always wins** WordPress’s standard Apache rules only route a request to `index.php` when the requested path is not an existing file (`RewriteCond %{REQUEST_FILENAME} !-f`). Upload a real `robots.txt` to the web root and the server returns it directly. WordPress never runs, so the `robots_txt` filter, core sitemaps and any plugin editor that relies on the filter stop affecting what crawlers read.

## Where is robots.txt in WordPress?

Crawlers only ever request the file at the root of a host: `https://example.com/robots.txt`. Google’s robots.txt specification states that a file in a subdirectory has no effect, and that each subdomain, protocol and port needs its own file. On a WordPress site that leaves two possible sources:

- **A physical file** in the web root, the same folder that holds `wp-config.php` on a standard install. Your host’s file manager or an SFTP client shows it.
- **The virtual file** WordPress generates when that folder has no `robots.txt`. It never appears in a file listing, which is why people search for it and find nothing.

To see what crawlers actually receive, request the live URL from outside your own network. Opening it in a browser works; a command-line request also shows the headers and whether a cache answered:

View your live robots.txt and its headers

```bash
# The body crawlers read
curl -s https://example.com/robots.txt

# Status, content type and cache headers
curl -sI https://example.com/robots.txt

# Is there a physical file? Run on the server, from the web root
ls -la robots.txt
```

## The default WordPress robots.txt

On a public site with pretty permalinks and no SEO plugin, current WordPress core returns this:

robots.txt: WordPress core default

```text
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml
```

| Line | Where it comes from | What it does |
| --- | --- | --- |
| `User-agent: *` | `do_robots()` | Rules for every crawler without a group of its own. |
| `Disallow: /wp-admin/` | `do_robots()`, built from `admin_url()` | Keeps crawlers out of the dashboard. |
| `Allow: /wp-admin/admin-ajax.php` | `do_robots()` | Keeps the AJAX endpoint fetchable for front-end features. |
| `Sitemap: …/wp-sitemap.xml` | Core sitemaps, via the `robots_txt` filter | Points crawlers at the sitemap index, on public sites only. |

The sitemap line comes from the `WP_Sitemaps` class, which hooks `robots_txt` and appends the index URL only when the public flag is true. With pretty permalinks the index is `/wp-sitemap.xml`; with plain permalinks it is `/?sitemap=index`. Core sitemaps are on by default for public sites and can be switched off with the `wp_sitemaps_enabled` filter.

> **Yoast changes the default** Yoast SEO’s XML sitemaps replace the core sitemap with `sitemap_index.xml`. Its developer documentation describes a default block wrapped in `# START YOAST BLOCK` and `# END YOAST BLOCK` comments, containing `User-agent: *`, an empty `Disallow:` and a `Sitemap:` line for `sitemap_index.xml`, placed ahead of any existing rules. If your file shows that block, Yoast is writing it.

## What “Discourage search engines” does to robots.txt

Settings → Reading has a **Search engine visibility** checkbox labeled “Discourage search engines from indexing this site.” It stores `blog_public` as false. Older WordPress versions answered robots.txt with `Disallow: /` when it was checked. Current versions leave robots.txt rules alone and add a `noindex, nofollow` robots meta tag to every page instead.

- **The robots.txt body still carries the default admin rules.** Core no longer prints `Disallow: /` for a private site.
- **The sitemap line disappears.** Core sitemaps are disabled for non-public sites, and `add_robots()` skips the line.
- **Every page gets a noindex tag**, through the `wp_robots` filter in the page head.
- **It is a request, not a lock.** WordPress’s own documentation says the setting does not block access and that it is up to search engines to honor it.

The change is deliberate: a crawler has to fetch a page to see its `noindex` tag, and Google warns that a URL disallowed in robots.txt can still be listed from links elsewhere. To keep a staging site away from every bot, put it behind HTTP authentication.

## How to edit robots.txt in WordPress

Pick one method. Most broken WordPress robots.txt files come from two fighting: an old physical file silently overriding the plugin editor someone uses today.

| Method | Where to find it | What it changes |
| --- | --- | --- |
| **Physical file** | Your host’s file manager, or SFTP to the web root | Replaces the virtual file entirely. WordPress and plugins no longer touch the output. |
| **Yoast SEO** | Yoast SEO → Tools → File editor, then **Create robots.txt file** | Yoast’s editor for the site’s robots.txt. Yoast notes the menu is hidden when file editing is disabled in WordPress. |
| **Rank Math** | Rank Math SEO → General Settings → Edit robots.txt | A virtual file managed from the dashboard. Rank Math notes a physical file in the root can stop the editor working. |
| **All in One SEO** | All in One SEO → Tools → Robots.txt Editor, then turn on **Enable Custom Robots.txt** | Adds rules to the virtual file WordPress generates. Rules are stored in the database, not on disk. |
| **Code** | The `robots_txt` filter in a small plugin or must-use plugin | Appends or rewrites the virtual output. Version-controlled and survives theme changes. |

Plugin menus move between releases, so if a path above does not match your dashboard, check the plugin’s own help pages. For the code route, a must-use plugin in `wp-content/mu-plugins/` keeps the rule active whatever theme or plugins are installed:

wp-content/mu-plugins/robots-rules.php

```text
<?php
/**
 * Plugin Name: Robots.txt rules
 */
add_filter( 'robots_txt', function ( $output, $public ) {
	if ( ! $public ) {
		return $output; // Leave private sites alone.
	}

	$output .= "\nUser-agent: GPTBot\nDisallow: /\n";

	return $output;
}, 10, 2 );
```

The filter only runs for the virtual file, so delete or rename any physical `robots.txt` first. If you prefer a static file, write all of it yourself, including the admin rules and sitemap URL, because nothing will add them.

## What not to block in WordPress robots.txt

Old tutorials recommend disallowing core folders. Search engines render pages and need the files that build them, and Google says not to block resources when their absence makes a page harder to understand.

- **`/wp-content/`, `/wp-content/themes/` and `/wp-content/plugins/`.** They hold the CSS and JavaScript your pages render with. Block them and crawlers see an unstyled, partly empty page.
- **`/wp-includes/`.** Core’s front-end scripts load from here. The same rendering problem applies.
- **`/wp-content/uploads/`.** Your images and media. Blocking it keeps them out of image search and away from any assistant that would show or cite them.
- **`admin-ajax.php`.** Leave the default `Allow` in place. Infinite scroll, filters and some product variations load through it.
- **Your whole site, by accident.** A `Disallow: /` copied from staging is the costliest line in robots.txt. Check the live file after every migration.

Paths worth disallowing generate endless or private URLs: internal search results, filter combinations and, on WooCommerce, the cart, checkout and account pages.

robots.txt: a lean WordPress and WooCommerce baseline

```text
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/

Sitemap: https://example.com/wp-sitemap.xml
```

## Adding AI crawler rules to WordPress robots.txt

AI operators publish separate robots.txt tokens for separate jobs: one for collecting training data, one for building an AI search index, and one for fetching a page live when a person asks about it. That split lets a WordPress site decline training while staying readable to AI search and assistants. Each vendor documents its own tokens:

| Token | Operator | Job | Follows robots.txt |
| --- | --- | --- | --- |
| `GPTBot` | OpenAI | Content that may be used to train its foundation models | Yes |
| `OAI-SearchBot` | OpenAI | Surfacing sites in ChatGPT search | Yes |
| `ChatGPT-User` | OpenAI | Fetches a page for a user’s request | OpenAI says robots.txt rules may not apply |
| `ClaudeBot` | Anthropic | Content that may be used for training | Yes |
| `Claude-SearchBot` | Anthropic | Indexing for Claude’s search results | Yes |
| `Claude-User` | Anthropic | Fetches a page when a Claude user asks | Yes |
| `PerplexityBot` | Perplexity | Surfacing and linking sites in Perplexity search; not used for training | Yes |
| `Google-Extended` | Google | A control token, never a crawler: whether Google may use crawled content for Gemini | Yes; does not affect Google Search |

Here is a WordPress file that declines training and keeps AI search and user fetches open. Notice the admin rules are repeated inside the named group:

robots.txt: WordPress with AI crawler rules

```text
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# Decline model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# AI search and user-requested fetches
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml
```

> **A named group drops your wildcard rules** A crawler obeys only the most specific group that names it. Once `OAI-SearchBot` has a group of its own, it ignores every line under `User-agent: *`, including your admin and checkout rules. Copy the paths you still want blocked into each named group.

Build these groups with the free [AI robots.txt generator](/tools/robots-txt-generator), which copies your private paths into every crawler group, then paste the result into your SEO plugin’s editor or a physical file. For user-agent strings and verification, see the [AI agent user agents reference](/learn/ai-agent-user-agents) and the guides to [ClaudeBot](/learn/claudebot), [PerplexityBot](/learn/perplexitybot) and [GPTBot](/solutions/gptbot).

robots.txt is a request that well-behaved crawlers honor. Where a block has to hold, add a firewall or CDN rule too.

## Multisite, subdirectory installs and server configuration

WordPress only registers its robots.txt rewrite when the site’s home URL is the root of the host. Core’s rewrite code says so in a comment: “only if installed at the root.” That one condition explains most of the edge cases:

- **WordPress in a subdirectory** (home URL `example.com/blog/`): no virtual robots.txt. Crawlers read `example.com/robots.txt`, which belongs to whatever serves the root, so put a physical file there.
- **Subdomain multisite** (`shop.example.com`, `blog.example.com`): each site is at the root of its own host, so each gets its own virtual file with its own visibility setting.
- **Subdirectory multisite** (`example.com/shop/`): every site shares one host, so crawlers read one robots.txt, the main site’s. Rules for a sub-site go in that shared file, written as paths such as `Disallow: /shop/checkout/`.
- **A physical file on a network**: if every site shares one web root, a physical `robots.txt` is served to every host in the network, overriding every site’s virtual file at once.

Server configuration can also stop the request before WordPress sees it. If `/robots.txt` returns a not-found page on a site with no physical file, check that pretty permalinks are on and that the server sends unknown paths to `index.php`. On Nginx, an exact-match block for robots.txt, like the one in the global restrictions example of WordPress’s own Nginx guide, serves only a physical file unless it includes a `try_files` fallback:

nginx: let WordPress answer robots.txt

```text
location = /robots.txt {
    allow all;
    log_not_found off;
    access_log off;
    try_files $uri /index.php?$args;
}
```

## Caching and CDNs serving a stale or altered robots.txt

You changed the rules, the editor shows the new version, and crawlers still read the old one. Work outward from WordPress:

1. **Check for a physical file.** It overrides every plugin editor and the `robots_txt` filter.
2. **Purge your page cache.** A page cache, in a plugin or at your host, can store the robots.txt response like any other URL.
3. **Purge the CDN.** Cloudflare’s documentation states that it caches a site’s robots.txt by default, even though plain text files are not otherwise on its default cache list. Purge the URL after every change.
4. **Look for rules the CDN adds.** Cloudflare’s managed robots.txt, when Bot Preference Sync is on and an AI policy is set to block, prepends a Content Signals block and `Disallow` rules for AI crawlers ahead of your own file. Those lines are not in WordPress.
5. **Allow for crawler caching.** Google says it generally caches robots.txt for up to a day, and may keep it longer if your server returns errors. A correct file can take that long to take effect.

Finally, request the file from outside your network without cookies. A security plugin or firewall that challenges unknown clients can return a challenge page instead of plain text.

## Serving AI agents on WordPress with the Rebilder plugin

robots.txt decides which bots may fetch a page, not what they read once they do: often a theme’s menus, sliders and cookie banners wrapped around the price. The free [Rebilder WordPress plugin](/docs/adapters/wordpress) handles that second part on your own server.

- **Markdown for agents, same URL.** A request with `Accept: text/markdown`, or from an identified agent such as `ChatGPT-User` or `ClaudeBot`, gets the post, page or WooCommerce product as markdown. Products include name, price as your shop displays it, availability, stock, SKU, categories and attributes.
- **People and search engines are untouched.** Browsers get your normal page. Googlebot and bingbot are classified first and always get the HTML a person gets, even if they ask for markdown.
- **`/llms.txt`.** An index of your site’s content, so an agent landing on your domain can find what is there.
- **`/answers`, only if you import it.** A Q&A set exported from your Rebilder account, served as markdown to agents and HTML to people.
- **No requests until you connect it.** Out of the box the plugin makes no network request. Reporting to the Console starts only when you add an API key.

The plugin leaves robots.txt to you. It adds no `robots_txt` filter, writes no file, and its markdown handler returns early on robots.txt requests, so whatever WordPress, your SEO plugin or your physical file serves today is exactly what crawlers keep reading. Set your crawler rules with the methods above; the plugin makes sure the agents you allow get a clean answer.

## Where is the robots.txt file in WordPress?

On most sites it is not a file at all. WordPress generates a virtual robots.txt at `https://your-site/robots.txt` whenever the web root has no physical file. If you or your host uploaded one, it sits in the web root next to `wp-config.php` and replaces the virtual version.

## What is the default WordPress robots.txt?

A `User-agent: *` group that disallows `/wp-admin/`, allows `/wp-admin/admin-ajax.php`, and, on a public site, a `Sitemap:` line pointing to `/wp-sitemap.xml`. SEO plugins such as Yoast change this default.

## How do I edit robots.txt in WordPress without a plugin?

Upload a physical `robots.txt` to the web root over SFTP or your host’s file manager, which replaces the virtual file, or add a `robots_txt` filter in a must-use plugin to change the virtual output. Do not combine the two: the physical file wins and the filter stops running.

## Why are my robots.txt changes not showing?

Usually one of three things. A physical file is overriding your plugin editor, a page cache or CDN is serving an old copy, or your CDN is adding its own rules. Check for a file in the web root, purge both caches, and request `/robots.txt` from outside your network.

## How do I block AI crawlers in WordPress robots.txt?

Add a group per token, such as `User-agent: GPTBot` with `Disallow: /`, through your SEO plugin’s editor, a physical file or the `robots_txt` filter. Block training tokens like `GPTBot`, `ClaudeBot` and `Google-Extended` separately from search and user-fetch tokens, and repeat your private paths inside every named group.

## Does the Rebilder WordPress plugin change robots.txt?

No. It serves markdown to AI agents, `/llms.txt`, and an optional `/answers` directory, and it skips robots.txt requests entirely. Your crawler rules stay wherever you manage them today.

## Related

- [Free AI robots.txt generator](https://rebilder.com/tools/robots-txt-generator)
- [WordPress plugin setup](https://rebilder.com/docs/adapters/wordpress)
- [AI agent user agents: the full reference](https://rebilder.com/learn/ai-agent-user-agents)
- [ClaudeBot: user agent and robots.txt](https://rebilder.com/learn/claudebot)
- [PerplexityBot: user agent, robots.txt and IPs](https://rebilder.com/learn/perplexitybot)
- [Cloudflare AI Crawl Control](https://rebilder.com/learn/cloudflare-ai-crawl-control)
- [Free llms.txt generator](https://rebilder.com/tools/llms-txt-generator)