---
title: "robots.txt sitemap: how to add the Sitemap directive the right way"
description: "Add your sitemap to robots.txt: Sitemap directive syntax, multiple sitemaps, index files, Search Console and Bing submission, IndexNow and common mistakes."
canonical_url: https://rebilder.com/learn/robots-txt-sitemap
last_updated: "2026-11-15"
---

# [robots.txt sitemap: how to add the Sitemap directive the right way](https://rebilder.com/learn/robots-txt-sitemap)

> Add your sitemap to robots.txt: Sitemap directive syntax, multiple sitemaps, index files, Search Console and Bing submission, IndexNow and common mistakes.

- **Updated:** 2026-11-15
- **Author:** Rebilder
- **Published:** 2026-11-15

**A robots.txt sitemap line is a `Sitemap:` record in your `robots.txt` file that gives crawlers the full URL of your XML sitemap, so they can find every page you want discovered without waiting to stumble on links.** It takes one line, it applies to every crawler that reads the file, and Google, Bing and other major search engines support it.

This guide covers the exact syntax, where the line goes, how to list several sitemaps or a sitemap index, how the line relates to submitting in Google Search Console and Bing Webmaster Tools, where IndexNow fits, what WordPress and Shopify write for you, and the mistakes that quietly stop a sitemap from being read. To build a complete file with AI crawler rules and a sitemap line, use the free [robots.txt generator](/tools/robots-txt-generator).

## What is the Sitemap directive in robots.txt?

Your `robots.txt` file mostly tells crawlers where they may not go. The `Sitemap:` line does the opposite job: it points them at a list of the URLs you want crawled. A crawler that fetches `https://example.com/robots.txt` reads the line, fetches the sitemap it names, and adds those URLs to its queue.

robots.txt: the smallest useful file with a sitemap

```text
User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml
```

Three standards define the pieces. The robots exclusion standard covers the file itself, the sitemaps.org protocol defines the XML format and the `Sitemap:` line, and each search engine documents how it reads both. Keep their own pages to hand:

Primary references

```text
Robots Exclusion Protocol (RFC 9309):  https://www.rfc-editor.org/rfc/rfc9309.html
Sitemaps protocol:                     https://www.sitemaps.org/protocol.html
Google robots.txt spec:                https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt
Google sitemap guide:                  https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap
IndexNow:                              https://www.indexnow.org/documentation
```

The line is a discovery aid, not an instruction. Google calls a submitted sitemap “merely a hint”: it helps a crawler find URLs, and the crawler still decides what to fetch and index. A `Sitemap:` line never overrides a `Disallow` rule, and it never forces a page into an index.

## robots.txt sitemap syntax and where the line goes

The record is the field name, a colon and an absolute URL. The rules that matter in practice:

- **Use the full URL.** Include the scheme and host: `Sitemap: https://example.com/sitemap.xml`. Google requires a fully qualified URL, and the sitemaps.org protocol asks for the complete address. A relative path such as `Sitemap: /sitemap.xml` is the single most common error.
- **Put it anywhere in the file.** The sitemaps.org protocol says the line is independent of the `User-agent` line, so its position does not matter. Google says the field is not tied to any specific user agent and may be followed by all crawlers. Most sites put it at the end, after the last group, where people reading the file expect it.
- **It does not end a group.** The robots exclusion standard lists `Sitemap` among the “other records” a crawler may interpret, and says such a record must not terminate a group. A sitemap line placed between two rules leaves both rules in the same group.
- **The field name is case-insensitive.** `Sitemap:`, `sitemap:` and `SITEMAP:` all work. Write it as `Sitemap:` for readability.
- **The sitemap can live on another host.** Google says the sitemap does not have to sit on the same host as the `robots.txt` file. The sitemaps.org protocol treats a `robots.txt` reference as your proof that you own the URLs a cross-host sitemap lists.

robots.txt: groups first, sitemap lines last

```text
User-agent: *
Disallow: /cart
Disallow: /checkout/
Disallow: /account/

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml
```

> **The sitemap line does not belong to the group above it** Writing `Sitemap:` directly under `User-agent: GPTBot` does not make the sitemap GPTBot-only, and it does not hide it from other crawlers. Every crawler that reads sitemap records reads every one in the file.

## Multiple sitemaps and sitemap index files

You can list as many sitemaps as you need, one per line. Google sets no limit on the number of `sitemap` fields in a file. Large sites usually split by content type, so products, articles and store locations each have their own file and their own error report:

robots.txt: several sitemaps

```text
Sitemap: https://example.com/sitemap-products.xml
Sitemap: https://example.com/sitemap-articles.xml
Sitemap: https://example.com/sitemap-locations.xml
```

Each sitemap file has a size ceiling set by the protocol. Past it, split the file and tie the parts together with a sitemap index, then list only the index in `robots.txt`. The sitemaps.org protocol says listing the index is enough; you do not need to name each child sitemap as well.

sitemap_index.xml (limits per sitemaps.org and Google)

```text
<?xml version="1.0" encoding="UTF-8"?>
<!-- One sitemap: max 50,000 URLs and 50MB uncompressed.
     One index:   max 50,000 child sitemaps, same 50MB cap. -->
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-products-1.xml</loc>
    <lastmod>2026-11-14</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-products-2.xml</loc>
  </sitemap>
</sitemapindex>
```

Gzip is fine. The size limit applies to the uncompressed file, so compressing a sitemap saves bandwidth but does not let it hold more URLs.

Mind the path rule too. The protocol ties a sitemap to its location: a file at `/catalog/sitemap.xml` may list `/catalog/` pages but not pages under `/images/`. Google applies the same limit unless you submit the sitemap through Search Console. Keeping sitemaps at the root avoids the question entirely.

## What the sitemap file itself should contain

A sitemap is a UTF-8 XML file. Only `<urlset>`, `<url>` and `<loc>` are required; every other tag is optional, and the major engines treat them differently:

| Tag | Required | How Google treats it | How Bing treats it |
| --- | --- | --- | --- |
| `<loc>` | Yes | The canonical URL to crawl. | The canonical URL to crawl. |
| `<lastmod>` | No | Used when it is consistently and verifiably accurate. | Should reflect the true last change to the page content, with a timestamp. |
| `<changefreq>` | No | Ignored. | Ignored. |
| `<priority>` | No | Ignored. | Ignored. |

sitemap.xml: two URLs, honest dates

```text
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/products/trail-runner</loc>
    <lastmod>2026-11-12T09:30:00+00:00</lastmod>
  </url>
  <url>
    <loc>https://example.com/shipping-and-returns</loc>
  </url>
</urlset>
```

- **List canonical URLs only.** Leave out redirects, duplicates with tracking parameters, pages that return errors and pages carrying `noindex`. A sitemap full of URLs the engine will reject teaches it to trust your sitemap less.
- **Set `lastmod` from the content, not the build.** A date that changes on every deploy, whether or not the page did, is the kind of value both engines learn to discount. Leave the tag off a page that has no meaningful date.
- **Escape the URLs.** The protocol requires entity escaping, so an ampersand in a query string is written `&amp;`.

## Submit your sitemap to Google Search Console and Bing Webmaster Tools

The `robots.txt` line and a direct submission do different jobs, and most sites should do both. The line lets any crawler find the sitemap the next time it reads your file. A submission gets you a report: when the engine last read the sitemap, how many URLs it found and which ones failed.

1. **Google Search Console.** Open your property, go to the **Sitemaps** report, enter the sitemap or index URL and submit. The report shows when Googlebot last fetched it and any processing errors. Google also accepts submissions through the Search Console API.
2. **Bing Webmaster Tools.** Open the site, go to **Sitemaps** and use **Submit sitemaps**. Bing accepts XML, RSS, Atom and plain-text sitemaps, and its list shows sitemaps you submitted, ones imported from Google Search Console and ones Bing found on its own. Bing says it fetches a sitemap as soon as it is submitted and then revisits it at least once a day.
3. **Check the fetch.** After submitting, confirm the status reads as successful and the discovered URL count is close to what you expect. A count near zero usually means the file is returning HTML, a redirect or a login page.

> **Sitemap ping URLs no longer work** Google retired its unauthenticated sitemap ping endpoint, and Bing removed anonymous sitemap submission, both because spammers abused them. Old plugins that still ping do no harm, but they do nothing. Use the `robots.txt` line, the webmaster consoles or, for Bing and the other participating engines, IndexNow.

## IndexNow: a complement to your sitemap

IndexNow is a protocol for telling search engines that a specific URL was added, updated or deleted, at the moment it happens. Participating engines share submissions with each other, so one notification reaches all of them. The engines listed on indexnow.org are Amazon, Bing, Naver, Seznam, Yandex and Yep, along with an endpoint for the Internet Archive. Google is not on the list.

It does not replace a sitemap. The IndexNow FAQ recommends a sitemap for complete, ongoing discovery and IndexNow for pages that change often, and Bing’s webmaster blog gives the same advice: sitemaps for coverage, IndexNow for fast URL-level updates. A price change, a restock or a new product page is the kind of event IndexNow was built for.

IndexNow: notify participating engines of one changed URL

```bash
# 1. Host your key as a text file at the root:
#    https://example.com/<your-key>.txt  (file content: <your-key>)

# 2. Ping one URL
curl "https://api.indexnow.org/indexnow?url=https://example.com/products/trail-runner&key=<your-key>"

# 3. Or batch up to 10,000 URLs per POST
curl -X POST "https://api.indexnow.org/indexnow" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{"host":"example.com","key":"<your-key>","urlList":["https://example.com/products/trail-runner","https://example.com/products/ridge-pack"]}'
```

Neither channel guarantees indexing. The IndexNow FAQ says so plainly, and so does Google about sitemaps. Both shorten the time between your change and the crawler’s visit; the engine still decides what to keep.

## What WordPress, Shopify and SEO plugins already write

Check your platform before adding a line by hand. Many already generate both the sitemap and the `robots.txt` reference, and a second, conflicting line is a common source of confusion:

| Platform | Default sitemap | robots.txt line |
| --- | --- | --- |
| WordPress core | `/wp-sitemap.xml`, a sitemap index; `/?sitemap=index` with plain permalinks. | Added to the virtual `robots.txt` on sites visible to search engines. See [WordPress robots.txt](/learn/wordpress-robots-txt). |
| Yoast SEO on WordPress | `/sitemap_index.xml`, replacing the core sitemap. | Yoast writes its own block with a `Sitemap:` line for its index. |
| Shopify | `/sitemap.xml` at the root of each store domain, updated as you add products, collections, pages and posts. | The theme’s `robots.txt.liquid` template prints the sitemap line from Shopify’s default groups. Add extra `Sitemap:` lines outside that loop. |
| Static site or custom stack | Whatever your build or framework generates. | Write the line yourself, with the absolute production URL. |

On WordPress, a physical `robots.txt` uploaded to the web root replaces the virtual file, and the core sitemap line goes with it. If the line vanished after you uploaded a file, add it back yourself.

## Common robots.txt sitemap mistakes

Most broken sitemap setups fail silently: the file looks right, and the engine reports nothing until someone checks. These are the faults to look for first:

- **A relative or wrong-host URL.** After a move to HTTPS or a new domain, the line still points at `http://` or the old host. Update it in the same change as the migration.
- **The sitemap is disallowed.** A broad rule such as `Disallow: /*.xml` blocks the sitemap itself. Google notes the sitemap field is followed provided the sitemap is not disallowed.
- **The sitemap lists blocked or noindex pages.** Telling an engine to crawl a URL your own rules forbid creates errors in its report and mixed signals about which pages matter.
- **robots.txt returns a server error.** Under the robots exclusion standard, a server error on `robots.txt` means the crawler must assume it may fetch nothing. Your sitemap line is unreadable, and so is the rest of your site.
- **A firewall or bot challenge in front of the files.** If your CDN serves a challenge page to crawlers, they receive HTML instead of your rules and your sitemap. Fetch both from outside your network with a crawler user agent and confirm plain text and XML come back.
- **Staging rules shipped to production.** A `Disallow: /` copied from a staging site blocks everything regardless of how good the sitemap is.
- **One sitemap line for several subdomains.** Every host serves its own `robots.txt`. `shop.example.com` needs its own file and its own sitemap reference.
- **Stale caches.** Engines cache `robots.txt`; Google generally keeps a copy for up to a day. A fix takes effect after the next fetch, not the moment you save.

Check the files a crawler actually receives

```bash
# robots.txt: expect 200 and text/plain, and your Sitemap line
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/robots.txt
curl -s https://example.com/robots.txt | grep -i "^sitemap:"

# The sitemap: expect 200 and an XML content type, not text/html
curl -sI https://example.com/sitemap.xml
curl -s https://example.com/sitemap.xml | head -5
```

Prefer a browser to a terminal? The [robots.txt checker](/tools/robots-txt-checker) reads a pasted or fetched file, lists the sitemap lines it finds and flags any that are not absolute URLs.

## Sitemaps, AI crawlers and Rebilder

Because the `Sitemap:` record is not tied to a user agent, any crawler that reads your `robots.txt` can see it, AI crawlers included. Whether a given AI crawler uses sitemaps for discovery is up to its operator, so check that operator’s documentation rather than assuming. Amazon is one documented case: it runs an IndexNow endpoint, so a change you announce through IndexNow reaches Amazon too.

Rebilder works on the other side of discovery: what an AI agent reads once it arrives. Here is where each part fits with your sitemap:

- **Your `sitemap.xml` stays yours.** The gateway does not write it. Set `<lastmod>` from the date each page states about itself, as [Help agents find your markdown](/docs/discovery) describes, so the date in your sitemap and the date an agent reads agree.
- **A markdown sitemap for agents.** The gateway can publish `/sitemap.md`, a markdown list of your pages built from your configured product, page and policy sources, for agents that read lists more easily than XML.
- **Markdown at the same URL.** When a recognized AI agent requests a page your sources answer, the gateway returns markdown from those sources. Search crawlers such as Googlebot and bingbot always get your canonical HTML, so your sitemap URLs serve them exactly what they served before.
- **Free tools.** The [robots.txt generator](/tools/robots-txt-generator) takes an optional sitemap URL, checks that it is a full http or https address, and appends it as the last line. The [llms.txt generator](/tools/llms-txt-generator) builds a short, curated map of your key pages alongside it.

## How do I add a sitemap to robots.txt?

Add a line with `Sitemap:` followed by the absolute URL of your sitemap, for example `Sitemap: https://example.com/sitemap.xml`. It can go anywhere in the file, but the end is conventional. Use the full URL with `https://` and the host, not a relative path.

## Where should the Sitemap line go in robots.txt?

Anywhere. The sitemaps.org protocol says the line is independent of `User-agent` lines, and the robots exclusion standard says a sitemap record does not end a group. Putting it at the bottom of the file, after your groups, keeps the file easy to read.

## Can robots.txt list more than one sitemap?

Yes. Add one `Sitemap:` line per file; Google sets no limit on how many. If you use a sitemap index, listing the index alone is enough, because crawlers follow it to the child sitemaps.

## Do I still need to submit my sitemap in Search Console if it is in robots.txt?

The `robots.txt` line is enough for Google and Bing to find the sitemap. Submitting it in Google Search Console and Bing Webmaster Tools as well gives you reports on when it was read, how many URLs were discovered and which failed, which is how you catch problems.

## Does the Sitemap directive apply only to the user agent above it?

No. Google says the sitemap field is not tied to any specific user agent and may be followed by all crawlers. A `Sitemap:` line under a `User-agent: GPTBot` group is visible to every crawler that reads sitemap records.

## Is IndexNow a replacement for a sitemap?

No. IndexNow notifies participating engines, including Bing and Amazon, about individual URLs as they change, and the IndexNow FAQ recommends keeping a sitemap for complete discovery. Use both: the sitemap for coverage, IndexNow for fast updates. Google does not participate in IndexNow.

## Can my sitemap be hosted on a different domain?

Yes. Google says the sitemap does not have to be on the same host as `robots.txt`, and the sitemaps.org protocol treats a `robots.txt` reference as proof that you control the URLs a cross-host sitemap lists. Each host still needs its own `robots.txt` pointing to it.

## Why does Search Console say my sitemap could not be read?

Usually the URL returns something other than XML: an HTML error page, a redirect, a login or a bot challenge from your CDN. Fetch the sitemap URL directly, confirm a successful status and an XML content type, and check that no `Disallow` rule matches it.

## Related

- [Free AI robots.txt generator](https://rebilder.com/tools/robots-txt-generator)
- [WordPress robots.txt: defaults and edits](https://rebilder.com/learn/wordpress-robots-txt)
- [Help agents find your markdown](https://rebilder.com/docs/discovery)
- [Free llms.txt generator](https://rebilder.com/tools/llms-txt-generator)
- [AI agent user agents: the full reference](https://rebilder.com/learn/ai-agent-user-agents)
- [bingbot: Microsoft’s crawler explained](https://rebilder.com/learn/bingbot)