Large Scale SEO

When I take on a large-scale SEO project, my first move is always to map the site architecture. That means identifying every distinct page type and how they connect via navigation. I start with a list of URL patterns from a crawl or a sitemap, then classify them into categories, subcategories, product pages, blog posts, and so on. For an e-commerce client with 200,000 product pages across 50 categories, I grouped URLs by category and ensured each product was reachable within 4 clicks from the homepage via a clear [website structure](/website-structure/). This hierarchy also formed the basis for my XML sitemap strategy: I created separate sitemap files for each category, each under 50,000 URLs, and submitted them in a sitemap index. This kept Googlebot focused on the most important sections.

Next, I fix crawl paths. I optimise the internal linking to concentrate authority on high-value pages and reduce orphan pages. For the same client, I audited the link graph and found that some subcategory pages were orphaned because the navigation only linked from the homepage via a dropdown that wasn't fully crawlable. I added breadcrumbs and contextual links from product pages back to their subcategories. I also checked that JavaScript-loaded navigation was accessible to Googlebot using [technical SEO](/technical-seo/) best practices.

Then I move to template-level fixes. I implement self-referencing canonical tags on all paginated series — not pointing all pages to page one, which is a common mistake. I also validate structured data using a script that checks every product template. In that client, I found 12% of product templates had missing or broken schema markup, which I fixed in bulk.

Finally, I set up automated monitoring: weekly crawl log analysis for 4xx errors, changes in crawl frequency, and indexation status. This catches problems before they compound.

One of the biggest traps in large-scale SEO is indexing everything you can crawl. Just because Googlebot can reach a URL does not mean it should be indexed. Filter pages, session IDs, sort parameters, and internal search results can create millions of low-value duplicates. I always use robots.txt to block crawl of parameterised URLs and noindex only after testing a small set.

A blanket noindex is dangerous. I've seen practitioners block entire directories only to discover they included important category pages with similar URL patterns. Instead, use pattern matching: block via robots.txt first to save crawl budget, and noindex only for pages that have zero organic traffic and no internal linking value. This is especially critical for faceted navigation — a site with 10 filters each with 5 options can generate 10^5 = 100,000 combinations. I recommend using [canonical tags](/canonical-tag/) to consolidate duplicates and avoid [duplicate content](/duplicate-content/) penalties.

Don't rely on manual audits. At scale, you cannot spot everything by hand. Automated monitoring of crawl logs, index status, and server errors is non-negotiable. I've seen teams miss a 500 error on a template that affected 40,000 pages because they only checked the homepage. Core Web Vitals also matter at scale: a 200ms increase in server response time on a template used by thousands of pages can cut crawl rate by 15%. Always test template changes on a sample before rolling out.

Another watch-out: pagination handling. I often see canonicals pointing all paginated pages to page 1. That's wrong. Use self-referencing canonicals on each page and let Google decide which to index.

I used to think large-scale SEO was about metadata and keywords. I spent hours optimising title tags and meta descriptions page by page, thinking that would move the needle. It didn't. The real leverage came from architecture and crawl efficiency. When I finally mapped the site structure for a client with 500,000 pages, I found that 30% of crawled URLs were duplicates from faceted navigation. Fixing that saved 40% of crawl budget and increased indexation of core product pages by 22%.

I also underestimated how quickly duplicate URLs snowball. A few filter combinations can create millions of near-identical pages. I once implemented a blanket noindex on all /filter/ URLs, but it also caught /filter/best-sellers/ which were important landing pages. Now I always analyse URL patterns first, using a [robots.txt](/robots-txt/) block for crawl control and noindex only after pattern analysis. I also learnt that XML [sitemaps](/sitemap/) need to be separated by page type early on. Initially I dumped all URLs into one file, which caused slow processing and partial ingestion.

Another mistake: I ignored the impact of slow templates. I assumed that individual page speed was the focus, but a 200ms increase in database query time on a template used by 50,000 pages caused a 15% drop in Googlebot crawl rate within a week. Now I prioritise template-level performance and set up automated performance monitoring.

I still sometimes forget to validate structured data across all templates after a deployment. I catch it now because I have a script that runs daily.