SEO Indexing

Think of indexing like a library cataloging a new book. Crawling is the librarian walking the aisles and picking up a book. Indexing is reading it, writing a summary, and placing it on the shelf so someone can find it later. If the book never gets cataloged, no one can check it out — no matter how good it is.

I ran an audit last month for a client with 200 product pages. They assumed all were indexed because they had submitted a sitemap. Google Search Console showed only 150 were in the index. The missing 50 had thin descriptions — 15 words each — and no internal links from any other page. That is a concrete example of the gap between being crawled and being stored.

Indexing is not ranking. I have seen pages in the index for years without ever appearing in the top 50 results. Being indexed simply makes a page eligible. The index is a database, not a guarantee. Search engines use it to answer queries quickly instead of crawling the live web every time.

My rule of thumb: if a page is not indexed within 48 hours of publishing, something is wrong. The first check is Google Search Console's URL Inspection tool. If it says "URL is not on Google", the page was never crawled. If it says "Crawled but not indexed", the problem is quality or technical. I fix the quality first: one paragraph of 250 words with a specific claim beats three paragraphs of fluff for getting indexed.

Indexing matters most at three points in the lifecycle of a site.

Launching new content. I worked on a blog that published a guide on Monday and expected traffic by Wednesday. By Friday, zero clicks. The URL Inspection tool showed the page was "crawled but not indexed". The content was a rewrite of someone else's article with no original data or images. I added a unique chart, three internal links from high-authority pages, and a [301 redirect](/301-redirect/) from an old related page. Within 48 hours Google indexed it. The decision rule: new content gets a 24-hour index deadline; if it misses, I treat it as a content quality problem, not a technical one.

Site migrations or redesigns. Every URL change risks pages dropping out of the index. During a migration I handled, the client used a [canonical tag](/canonical-tag/) pointing to the new URL but forgot to update the [sitemap](/sitemap/). Google kept the old URLs indexed because the canonical was not supported by consistent signals. I resubmitted the new sitemap and removed the old URLs from it. Index coverage recovered in 10 days. The counter-argument: some people say canonicals are enough for indexing signals. I disagree. You need the sitemap, internal links, and redirects all pointing at the same target.

Troubleshooting traffic drops. If organic traffic halves overnight, check the index count in Search Console first. A client lost 40 % of traffic because a staging site with a [robots.txt](/robots-txt/) disallow for all was accidentally served to Googlebot. The production site was blocked. I removed the disallow, requested re-crawling for all key pages, and index coverage returned within a week. [Duplicate Content](/duplicate-content/) can also cause indexing issues — when Google sees two pages with identical content, it may pick one and exclude the other. I use the canonical tag to signal which version to index, not to hide the duplicate.

I used to think crawling and indexing were the same step. If Googlebot fetched a page, I believed it would be indexed automatically. That cost me months of delayed visibility for a client. Their product pages were crawled weekly but never indexed because they had no unique content — just manufacturer descriptions. I was blaming the server when the real problem was content depth.

I blocked important pages with robots.txt thinking I was saving crawl budget. The idea was that if I disallowed low-value pages, Googlebot would spend more time on high-value ones. Instead, I made those pages invisible to the index entirely. A robots.txt disallow prevents crawling, which prevents indexing. The only way to control indexing without blocking crawling is the noindex meta tag or the X-Robots-Tag header. Now I use robots.txt only for pages I never want indexed, like admin directories or staging environments.

I submitted every page to the sitemap, including thin or duplicate content. Google ignored the low-quality pages and sometimes devalued the sitemap as a whole. Now I only include pages that I can justify with a one-sentence answer to "why should this be in the index?" If a page has under 100 words of original text and no internal links, it does not go in the sitemap. I also check for [JavaScript SEO](/javascript-seo/) issues — pages loaded dynamically may not be indexed if Google cannot render them. I pre-render key content server-side or use dynamic rendering for JavaScript-heavy sites.

One more thing I got wrong: I assumed that once a page was indexed it stayed indexed. Indexing is dynamic. Google can drop a page if it finds a better result, if the content goes stale, or if the site loses authority. I now monitor index coverage monthly using [SEO Audit](/seo-audit/) tools and react to sudden drops by refreshing content and re-submitting the URL.