Log File Analysis

I start by locating and downloading the raw server logs. On a typical Apache or Nginx setup, that means finding the access.log files in /var/log/ or through the hosting control panel. For cloud-hosted sites, I check CloudFront logs on AWS, or the equivalent in Azure or GCP. I pull at least 60 days of logs, not 30. Why? Because a single week can't show you weekend versus weekday crawl patterns, and 30 days might miss a monthly bot cycle. With 60 days, I caught Googlebot hitting an old product feed URL exactly every 14 days — a pattern I would have missed with a shorter window.

Once I have the logs, I parse them into a structured format. I export to CSV and use a command-line tool like awk to filter for user agents containing 'Googlebot', 'Bingbot', etc. On one 200k-line log file, I found that Googlebot made 4,500 requests to a single blog archive page that returned 200 but had zero organic traffic. That's 4,500 wasted requests. I then grouped by status code: 85% 200, 8% 404, 5% 301, 2% 500. The 404s were mostly old removed pages with no redirect. I prioritised setting up [301 redirects](/301-redirect/) for those.

After that, I look at crawl frequency per URL segment. I use a pivot table to count requests per URL per day. Pages that get crawled 10 times a day but have never ranked are candidates for noindex or redirect. I also check for crawl loops: infinite spaces like parameter-based pagination that spin Googlebot around. I often find that [canonical tags](/canonical-tag/) or [robots.txt directives](/robots-txt/) can stop the waste.

My decision rule: if a page consumes more than 1% of total crawl budget but generates less than 0.1% of organic traffic, it needs fixing. In one case, a /filter/ page with 50,000 parameters ate 12% of the crawl budget. I blocked it in robots.txt and redirected the main variant to the category page.

The biggest mistake I see is treating logs like analytics. Logs record every HTTP request — from search engines, scrapers, malware scanners, and your own team. Analytics uses JavaScript and cookies to track user sessions. They are not interchangeable. I once saw a spike in Googlebot requests that turned out to be a security scanner spoofing the user agent. Always validate the IP against Google's official range.

Another watch-out: ignoring non-Google bots. Bingbot, Baiduspider, YandexBot, even DuckDuckBot all crawl your site. In some regions, Bingbot can account for 30% of search traffic. If you only filter for Googlebot, you miss crawl inefficiencies for those engines. I once found Bingbot hitting a paginated archive with 100 pages and no next/prev markup, causing [duplicate content](/duplicate-content/) issues. That wasted budget and risked penalties.

Don't forget to exclude your own IP addresses. Your team's monitoring tools, health checks, and even your own browsing can inflate log numbers. Add a filter clause to remove known office IPs. I learned this the hard way when I thought Googlebot was crawling every minute — it was our uptime check.

Finally, 200 status does not mean clean. A 200 on a page that should be a 404 or a redirect is a crawl budget leak. Similarly, a 200 on a page with noindex meta is fine, but if the page is being crawled frequently, that's still waste. Use log analysis to find those mismatches and fix them.

Some SEOs argue that log analysis is too technical and time-consuming for most sites. I disagree. Even a basic parse with free tools reveals more than Search Console's crawl stats. You don't need a full-time data engineer — just a weekend afternoon and a CSV. Log analysis is a core part of any [technical SEO](/technical-seo/) audit I run.

For years I treated log files as a sysadmin concern. I assumed that if the site was loading fast in a browser, crawling was fine. That was wrong. The first time I ran a proper log analysis, I found that Googlebot was spending 40% of its crawl budget on a set of PDFs that hadn't been updated since 2015. Those PDFs had 200 status, but they were linked from a sitemap that never got removed. I should have 301-redirected them to the current pages or removed them from the sitemap.

I used to think that all 200 responses were good. Then I found Googlebot hitting a URL that returned 200 but the page was actually a server-side error page that returned a 200 status instead of 5xx. That's called a soft 404. The server was misconfigured, and Google wasted thousands of requests on broken pages. Log analysis flagged this immediately.

I forgot to exclude my own office IP. For months, my team's health-check script, running every 30 seconds, looked like a Googlebot crawl spike. I was falsely alarmed that we were being over-crawled. Once I filtered out that IP, the real crawl rate was half of what I thought.

I also ignored Bingbot for years. I only filtered Googlebot. When I finally expanded the filter, I saw Bingbot crawling our /search/ URL with infinite query parameters, generating thousands of near-duplicate pages. I had to add a robots.txt directive to block /search?q= and set up proper canonical tags. That mistake cost months of wasted budget. I now run log analysis monthly as part of a regular [SEO audit](/seo-audits/), not just reactively.