Crawl Google

Crawling is the step where Googlebot downloads the content of a page. It is not the same as indexing. A page can be crawled but not indexed, or indexed without being crawled recently. The key is that Google must be able to discover your URLs in the first place. That happens through links, sitemaps, and your site structure. If you block Googlebot with a [robots.txt](/robots-txt/) directive, it will not even attempt to download the page. If your internal links are hidden inside JavaScript click handlers, Googlebot may never see them. I have seen sites with beautiful content that simply never got crawled because the navigation required a hover event. The solution is boring but effective: use standard <a href="..."> links. Submit a [sitemap](/sitemap/) to give Google a hint, but do not treat it as a guarantee. Sitemaps are suggestions, not orders. Google decides what to crawl based on its own algorithm, which prioritises pages that are linked from other trusted pages. For most sites, getting the basics right — crawlable links, clean robots.txt, no disallowed pages — solves 90% of crawl problems. The rest is fine-tuning.

Crawl budget — the number of URLs Googlebot will crawl on your site per day — only becomes a real concern when you have more than a few thousand URLs and Googlebot is not covering the important ones. For a small blog or a standard business site, crawl budget is rarely the bottleneck. The bottleneck is almost always poor [technical SEO](/technical-seo/): broken links, redirect chains, or pages that require JavaScript rendering without proper fallback. I have audited sites where Googlebot was wasting its daily crawl allowance on session IDs, filter parameters, and infinite calendar pages. That is when crawl budget matters. The fix is to use a robots.txt to block unimportant parameters, use canonical tags to consolidate duplicates, and structure your [website structure](/website-structure/) so that the most important pages get the most internal links. If you have a site with 50,000 product pages and Google crawls only 200 per day, you need to decide which 200 matter most. That is a crawl budget problem. But never optimise for crawl budget until you have confirmed that Google can actually reach your core pages. First, check the Index Coverage report in Search Console. If you see "Crawled - currently not indexed" for your money pages, you have a content quality or rendering issue, not a budget issue.

For the first two years of my career, I thought that submitting a sitemap was the primary way to get Google to crawl pages. I would spend hours generating perfect XML sitemaps, only to wonder why my new pages took weeks to appear. I had missed the obvious: Googlebot discovers most pages through links, not sitemaps. A sitemap is a safety net, not the main path. I also assumed that if a page was in the sitemap, it would be crawled. It will not be crawled if the page is blocked by robots.txt, returns a 5xx error, or has no inbound links from the rest of the site. The second mistake I made was treating crawl budget as a universal problem. I read blog posts about large e-commerce sites and tried to apply the same tactics to a 200-page brochure site. That was a waste of time. The third mistake was ignoring the impact of JavaScript SEO. I built a site with a React-based navigation and assumed Googlebot would handle it. It did not. The pages were technically crawlable — Googlebot downloaded the HTML — but the links were only rendered client-side, and Googlebot's resources were limited. I learned to test with the URL Inspection tool in Search Console, which shows exactly what Googlebot sees. That tool is now the first thing I open on any new audit.