A sitemap for a large website stops doing its job once the number of URLs passes 50,000 or the file grows past 50MB - that is a hard limit of the sitemaps.org protocol, not a recommendation. The fix is a sitemap index: a pointer file that references several child sitemap files, each staying within the limit.
Across Exceltic.dev projects we regularly see the same pattern: a site with a catalog of 80,000-150,000 product pages, or a blog with several hundred articles, is still running on a single sitemap.xml left over from early development. The file is technically valid until it hits the limit - but in practice Google Search Console has already been showing incomplete coverage for a while, and nobody connects it back to the sitemap.
Below: how a sitemap index is structured, how to split URLs across files for proper indexing monitoring, and which of the lastmod, changefreq, and priority attributes Google actually uses in practice, and which it ignores entirely.
Growth of a content site - a catalog, a blog with hundreds of posts, or multi-country pages - rarely feels like a single event. The sitemap limit gets crossed quietly, somewhere between two ordinary development sprints. The problem doesn’t surface as a build error - it surfaces as coverage in Google Search Console suddenly no longer growing along with the number of published pages.
When a single sitemap.xml stops being enough
An XML sitemap is a file in XML format listing a site’s URLs, telling search engines which pages exist and when they were last updated.
The sitemaps.org protocol sets a precise limit: no more than 50,000 URLs and no more than 50MB uncompressed per file. This limit applies even if the file is served compressed via gzip - the limit is calculated on the uncompressed size.
In practice, the size limit hits before the URL-count limit. If the sitemap contains long URLs with parameters, <image:image> tags, or multiple <lastmod> entries for hreflang versions, you can hit 50MB at as few as 30,000-35,000 entries.
Situations where the limit is almost unavoidably exceeded:
- A marketplace or e-commerce product catalog with 50,000+ SKUs.
- A blog or knowledge base that accumulates hundreds of articles over several years and adds tag and category pages on top.
- A multi-country or multilingual site where every page is duplicated across 3-5 language versions.
If a site has crossed the limit and sitemap.xml keeps listing every URL in sequence, Google will either cut the file off at the 50,000th entry or refuse to process it at all. Some pages drop out of the crawler’s view with no error message in any interface - the only sign is that coverage in Search Console stops growing.
What a sitemap index is and how it’s structured
A sitemap index is a file that doesn’t contain page URLs itself - it lists links to other sitemap files, each of which stays within the 50,000 URL and 50MB limit.
The structure is simple: the root file (usually sitemap-index.xml or sitemap.xml) contains a list of <sitemap> tags, each with the address of a child file and the date it was last modified.
Example sitemap index structure
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://example.com/sitemap-posts.xml</loc>
<lastmod>2026-08-20</lastmod>
</sitemap>
<sitemap>
<loc>https://example.com/sitemap-products-1.xml</loc>
<lastmod>2026-08-25</lastmod>
</sitemap>
<sitemap>
<loc>https://example.com/sitemap-products-2.xml</loc>
<lastmod>2026-08-25</lastmod>
</sitemap>
<sitemap>
<loc>https://example.com/sitemap-pages.xml</loc>
<lastmod>2026-07-01</lastmod>
</sitemap>
</sitemapindex>The sitemaps.org protocol sets the same limit for the index itself: up to 50,000 links to child sitemap files, with the index size capped at 50MB. In practice this is enough even for sites with tens of millions of pages, since each link in the index points to a separate file of up to 50,000 URLs.
Google and other search engines crawl the index recursively: they read the root file first, then each child sitemap as a regular URL list. For the crawler this is transparent - the difference only shows up in how you monitor the results.
How to split URLs across files meaningfully
The most common mistake is mechanically slicing the URL list into chunks of 50,000, without regard to content type. Technically this satisfies the limit, but it strips away the main benefit of a sitemap index - separate indexing monitoring by site section.
Google Search Console shows indexing statistics for each individual sitemap file. If you split URLs by content type - articles separately, products separately, static pages separately - you can immediately see which section is indexing worse than the rest.
Example: a site has 600 blog articles and 120,000 product pages in its catalog. A sensible split:
sitemap-posts.xml- all 600 articles, one file, nowhere near the limit.sitemap-products-1.xml…sitemap-products-3.xml- products at 40,000 URLs per file, with headroom below the 50,000 limit.sitemap-pages.xml- static pages (about, contact, landing pages).
With this structure, Search Console shows: sitemap-posts.xml at 95% of pages indexed, sitemap-products-2.xml at 40%. The problem is localized instantly - without this split you would have to manually cross-check thousands of URLs from one combined list.
For multilingual sites, it makes sense to use a separate sitemap file for each language version of the catalog or blog - the same separation also simplifies working with hreflang markup and checking it.
Step-by-step generation
Generation during a static site build
On a static or headless stack (Astro, Next.js with static export), the sitemap is generated as part of the build, not by hand. A build-time script walks through content collections or CMS data, groups URLs by content type, and writes separate XML files plus a root sitemap index.
Ready-made integrations like @astrojs/sitemap cover the basic case - a single flat sitemap.xml. Once a site outgrows the limit and needs a split by content type, you need a custom build script or endpoint that produces multiple files and an index over them.
Automatic updates as content is added
A sitemap should never be updated by hand. Every deploy - via CI/CD or a webhook from a headless CMS - should rebuild the current URL list and overwrite the files.
Key point: the date in <lastmod> should come from a real content-update field (an updatedAt field in the CMS, or the date of the file’s git commit), not be stamped with the current build date for every page on every deploy.
Registering in Google Search Console and robots.txt
In Google Search Console, you only need to submit the address of the root sitemap index - not every child file individually. Google will crawl all the nested files on its own.
It’s also worth adding the sitemap index address as a line in robots.txt:
Sitemap line in robots.txt
Sitemap: https://example.com/sitemap-index.xmlThese are two different mechanisms with different jobs: sitemap.xml tells the crawler which pages exist, while robots.txt and llms.txt control which crawlers are even allowed onto the site. One file without the other doesn’t add up to a full visibility strategy - a sitemap without a correct robots.txt can point to pages that robots.txt itself blocks from being crawled.
lastmod, changefreq, priority - what Google actually uses
Of the three optional sitemap attributes, Google by its own admission uses essentially only one - and not even always.
changefreq (the stated update frequency of a page) has been officially ignored by Google for years. You can leave the field in the file for compatibility with other systems, but it has no effect on indexing.
priority (a page’s relative priority from 0 to 1) is also not factored in by Google when deciding what and when to index. Search engines determine a page’s priority from internal linking and other signals, not from a value you assigned to yourself.
lastmod is the only field Google can genuinely make use of, but only under one condition: the value has to be accurate and reflect a real content change. If lastmod gets stamped with the current date on every rebuild for every page - including ones where nothing changed - Google eventually stops trusting the field altogether, across the entire domain.
The practical takeaway: don’t spend time configuring changefreq and priority - they’re dead weight that adds to file size without any benefit. Only set lastmod if it’s pulled from a real content-change date, not the build date.
Common mistakes
- A single sitemap.xml with 80,000+ URLs. Formally exceeds the protocol limit - Google simply won’t see some of the pages.
- lastmod = deploy date for every page. Damages trust in the field for the entire domain, not just the affected pages.
- The sitemap index isn’t listed in robots.txt and isn’t added to Search Console. The files exist on the server, but the search engine doesn’t know about them.
- Pages with noindex, redirects, or non-canonical duplicates end up in the sitemap. Such URLs burn crawl budget and create conflicting signals for Google.
- Splitting alphabetically or by add order instead of by content type. Technically stays within the limit, but makes section-by-section indexing monitoring pointless.
Who this actually matters for
A sitemap index isn’t needed for every site. A landing page or corporate site with 20-40 pages does perfectly well with a single sitemap.xml - splitting it into multiple files would only add maintenance overhead with no benefit.
This matters for content catalogs (thousands to tens of thousands of product pages), blogs and knowledge bases with hundreds of articles, and multi-country or multilingual sites. If an existing site still runs on a single sitemap.xml and its content volume has long outgrown that setup, adding a proper sitemap index structure to an already-running site is a targeted fix, not a reason to rebuild the whole project. If you’re instead planning a new content site with catalog or blog growth in mind, it’s worth designing the sitemap architecture from day one - it’s part of the work involved in building a corporate website from scratch.
Frequently asked questions
How many URLs can go in a single sitemap file?
No more than 50,000 URLs and no more than 50MB uncompressed - this is the official limit of the sitemaps.org protocol, not a recommendation from any particular search engine. Once you exceed the limit, you need a sitemap index that links to several separate files, each within the limit.
Can a sitemap be gzip-compressed?
Yes, a sitemap file can be served as .xml.gz, which reduces the amount of data transferred. Important: the 50MB limit is calculated on the file size before compression, not after. A sitemap that’s 45MB uncompressed still needs to be split, even if it’s only 4MB after gzip.
Do I need a sitemap index if my site has fewer than 50,000 pages?
There’s no formal requirement - a single sitemap.xml stays within the limit. But if the site mixes different content types (articles, products, static pages), splitting into multiple files via a sitemap index - even at just 5,000-10,000 URLs - gives you separate indexing monitoring by section in Google Search Console, which makes diagnosis easier.
Does a correct sitemap affect a site’s search rankings?
Not directly - a sitemap doesn’t push pages up in the results. But it does affect whether Google finds your pages at all, and how quickly it indexes them. Without a correct sitemap on a large site, some content can go years without ever entering the rankings, simply because the crawler doesn’t know it exists.
If your site’s sitemap hasn’t been reviewed in a while and your content volume has grown significantly, describe the situation to the Exceltic.dev team. We’ll review the current structure and propose a plan for rolling out a sitemap index sized to your content volume and type.