Crawlability

Crawlability is whether Googlebot can physically reach a page. Full stop. If the bot cannot make a successful HTTP request and retrieve the HTML (plus any critical resources), the page doesn't exist in Google's world. It never enters the index. I have seen people confuse crawlability with indexability all the time—a page can be crawled but blocked from indexing via noindex, or it can be indexable but unreachable. The simplest fix is to test a URL with Search Console's URL Inspection Tool and watch for 'URL is available to Google'. That green tick tells you the bot got there. If it is yellow or red, the barrier is almost always in robots.txt, a server error, or a broken link. For example, I audited a travel site that had accidentally blocked /js/ in robots.txt. Googlebot could not execute the JavaScript that loaded the main text content, so 40% of the pages rendered blank. The client had been wondering why their new blog posts never appeared in search. Once I allowed crawling of JS and CSS, the blog posts got indexed within a week. Crawlability is the gate; indexing is the room beyond.

Take an e-commerce site with 300,000 product pages. Googlebot has a crawl budget—roughly a few thousand URLs per day for a decent site. If the internal linking is a mess, with orphan categories and poorly prioritised product listing pages, Googlebot may crawl only the top 10,000 products each month. That leaves 290,000 products undiscovered indefinitely. Crawlability matters most when you have volume, but it also matters on small sites with poor navigation. I once helped a local bakery site that had added a 'seasonal menu' page, linked only from the footer. The footer itself was loaded via JavaScript and not crawled. That page never got picked up for six months. The fix was a simple HTML link in the main navigation. Another scenario: sites using infinite scroll without proper [pagination](/pagination/). Googlebot can't 'scroll', so if you don't provide proper crawlable links (like paginated URLs or load-more with permalinks), the bot only sees the first 'page'. I now apply this rule: if a page is not reachable within three clicks from the homepage via plain HTML links, it might as well be invisible. Counter-intuitively, some people think crawlability only concerns [robots.txt](/robots-txt/). It also includes HTTP response codes: a 301 redirect chain (A→B→C) wastes crawl budget. A 503 server error makes the bot retry later but can signal instability. A 404 on an important page severs the path entirely. So when you see a sudden crawl spike in Search Console, investigate the logs: often a new URL pattern has been discovered but returns errors, eating budget. Understanding the [website structure](/website-structure/) and using [technical SEO](/technical-seo/) basics like proper [canonical URLs](/canonical-url/) helps reduce wasted effort.

I used to think crawlability and indexability were the same. The moment a page got crawled, I assumed it would appear in search results. Then a client asked why their 'thank-you' page showed up in the index despite being noindexed. That taught me the distinction: crawling is the act of fetching; indexing is storing the content in the database. A page can be crawled but ignored by the index. Another mistake: I once blocked CSS and JS in robots.txt for a site, thinking it would speed up crawling—an old myth. Googlebot uses those resources to render the page. Blocking them can cause the page to appear broken or empty, impacting rankings. I learned that the hard way when a site's product descriptions never showed in search, until I allowed crawling of static assets. I also used to assume that all pages in an XML [sitemap](/sitemap/) would get crawled. In reality, Google uses the sitemap as a hint, not a command. On a site with 50,000 sitemap entries, Google may only crawl the ones it deems important based on its own algorithms—link equity, PageRank, recency. I changed my approach: instead of stuffing the sitemap with every page, I now include only canonical, indexable, high-value URLs. For less important pages (tag pages, filtered navigation), I let internal linking do the discovery—if those pages are valuable, they'll earn links. One more error: I used to ignore HTTP 500 errors as temporary blips. But if a page consistently returns a 503 or 500, Googlebot may reduce crawl frequency for the whole site. I now monitor server errors weekly, especially after code deployments, because a single misconfigured .htaccess can lock out entire directories. My understanding of [JavaScript SEO](/javascript-seo/) improved after that blocking mistake.