Sitemap

A sitemap is an XML file that lists the URLs you want search engines to consider for crawling and indexing. It is not a command—Google might still choose not to crawl a URL if it sees no value. But it is a strong signal, especially for pages that are deep in the architecture or have few internal links. I have seen sites with 10,000 blog posts where only 30% were discovered organically. After adding a proper sitemap and fixing internal linking, discovery hit 85%. That is not just the sitemap—it is the combination. Some SEOs treat the sitemap as a magic wand, ignoring link structure. I think that is lazy. Use the sitemap to hint at everything important, but also ensure those pages are linked from other content. One decision rule: include any URL that you would be disappointed not to find in Google's index. Exclude everything else – tag pages, parameterised URLs, internal search results. Also include lastmod if your CMS updates it accurately; Google uses it for freshness. Ignore changefreq and priority—Google has said they ignore those fields. For technical sites, a sitemap is one part of a broader [Technical SEO](/technical-seo/) strategy. It works alongside proper [pagination](/pagination/) handling to avoid duplicate content signals. I treat the sitemap as a curated list, not a full site dump.

A sitemap becomes critical when your site has more than 500 unique pages, when you launch a new site with few external links, or when you use JavaScript frameworks that delay content rendering. For JavaScript-heavy sites, a standard XML sitemap still works, but you may also need to consider [JavaScript SEO](/javascript-seo/) best practices to ensure dynamic routes are discoverable. I had a client with 200,000 product pages and a weak internal linking structure. Submitting a sitemap index with four sitemaps (50,000 each) made Google crawl 90% of those products within a week, compared to 20% before. Even for a 50-page site, a sitemap adds a safety net. Counter-argument: some argue that if your site is well-linked and small, a sitemap is redundant. I disagree—it is a low-effort optimisation that costs nothing and ensures you do not miss a page if a link gets broken. The decision rule: always have a sitemap, but prioritise which URLs go into it. Use a sitemap index if you have multiple sitemaps or exceed 50,000 URLs. If you serve content in multiple languages, include [hreflang tags](/hreflang-tags/) in the sitemap to signal regional variants. For sites relying on mobile traffic, ensure your sitemap covers both desktop and [Mobile SEO](/mobile-seo/) needs—Google uses mobile-first indexing, so the sitemap should reflect the mobile version.

Early on, I generated a sitemap from a crawler that included every URL it found: tag pages, paginated archives, filter combinations. I submitted it and wondered why Google was spending crawl budget on 50,000 near-duplicate URLs. My indexing rate was under 60%. When I stripped out everything except canonical product and category pages, that rate jumped to 85% in two weeks. I also forgot to update the sitemap after a site migration—Google kept hitting 301 redirects and 404s. Now I set up a weekly cron job that regenerates the sitemap and checks for errors. Another mistake: I did not include the Sitemap directive in [Robots.txt](/robots-txt/), assuming Search Console submission was enough. But if your robots.txt changes or Search Console loses the property, the sitemap might not be found. Now I always add Sitemap: https://yoursite.com/sitemap.xml to robots.txt. I also used to set changefreq='always' and priority='1.0' on every URL, thinking it would boost priority. Google's John Mueller confirmed they ignore those fields. I removed them and saw no change. The biggest lesson: a sitemap is a living file. Treat it like a product feed—curate, monitor, update. I also learned to use [canonical tags](/canonical-tag/) to handle duplicate content issues before they reach the sitemap. My current process: generate nightly, check Search Console for errors, remove any URL that returns a 4xx/5xx or is noindexed.