Proxy SEO

I start with the locations. If I am tracking rankings for "accounting software London" across 50 keywords, I need London IPs, not UK-wide. I pick residential proxies because datacenter proxies are too easily flagged by Google. I have run tests: a pool of 200 residential IPs from London postcodes gave me a 98% success rate on Google SERP scrapes, while a datacenter pool of the same size returned only 40% usable results – the rest were blocked or showed irrelevant rankings.

Once the proxy type is set, I configure rotation. I rotate the IP every 15 requests. That interval avoids pattern detection while keeping the pool alive for longer scraping sessions. I also throttle to 5 requests per second. Even with 200 IPs, sending 20 requests per second triggers rate limits on Google within minutes. I learned to back off by watching the HTTP 429 responses pile up in my logs.

For geo-targeted rank tracking, I use a proxy manager that lets me assign IPs by postcode sector. For example, I map each keyword to a specific London borough and only rotate within that borough's IP list. This gives me results that match what a user in that area would see. If you are scraping for a [technical SEO audit](/technical-seo/) across multiple regions, you need this level of granularity.

I also pair proxies with a headless browser that mimics a real device – desktop, mobile, or tablet – depending on the SERP type. Google serves different layouts and results based on [JavaScript SEO](/javascript-seo/) rendering and device type. I set the viewport and User-Agent to match the target market. I have seen a 20% increase in successful scrapes just by switching from a generic Python requests library to a browser-controlled scraper.

Finally, I monitor proxy health daily. I have an automated check that scrapes a known stable query every hour and compares the result to a manual browser search. If success rate drops below 90%, I replace the failing IPs. This costs me about £100 per month for a single client project, but the data quality justifies it. I would rather spend that than waste hours cleaning garbage data.

Free proxies are a trap. I tested a free list once and every single IP was either already blocked by Google or so slow that the scrape timed out. You end up spending more time debugging than collecting data. I only use paid residential proxy providers with a reputation for SEO work.

Ignoring rate limits is another mistake. Proxies spread your requests across IPs, but they do not grant immunity. If you send 100 requests per second across 100 IPs, each IP still hits the server once per second on average – enough to trigger rate limits after a few hundred requests. I throttle to 5 requests per second per IP and never exceed 10 overall. I also add random jitter between requests to avoid patterns.

Using the wrong proxy type costs you accurate data. For localised SERP tracking, datacenter proxies are useless because they are registered at data centres, not in the target city. Google often ignores the geo-location of datacenter IPs and returns a generic result. I learned this when a client in Berlin asked for local rankings and my datacenter pool showed results from Frankfurt. I switched to [programmatic SEO](/programmatic-seo/) proxies with residential IPs and the rankings matched manual checks.

Proxy pool size matters more than you think. If you rotate only 10 IPs across 1,000 keywords, each IP is reused too often and gets flagged quickly. I aim for at least 50 IPs per 100 keywords per location. For a large project, I use 300–500 IPs. This also helps with session persistence – some scraping tools need the same IP for a full page load. I make sure my proxy provider offers sticky sessions.

Another watch-out: cookie and header consistency. Even with residential IPs, if you send the same User-Agent, Accept-Language, and other headers from every IP, you create a fingerprint that can be blocked. I randomise these headers per request within a realistic range. I also use a dedicated browser profile for each proxy session to maintain cookies, which is critical when scraping pages that require a session token.

Finally, be careful with [web scraping](/web-scraping/) of Google's terms of service. I use proxies only for internal research and rank tracking, never for commercial redistribution of scraped data. I also respect robots.txt. If a path is disallowed, I skip it. This keeps my tooling on the right side of the policy.

I used to think any proxy would do. When I started out, I bought a cheap datacenter pool and tried scraping local SERPs for a client in Berlin. The results were wildly different from what I saw in a private browser window. I blamed the scraper for days before realising the datacenter IPs were all registered in Frankfurt, not Berlin. Google was returning results for Frankfurt users. I should have checked the IP geo-location before running the scrape. Now I always verify with a tool like whatismyipaddress.

I also assumed proxies alone would prevent blocking. I set up a residential pool, rotated well, and throttled correctly, but I still got blocked after a few hundred requests. The problem was I was sending identical HTTP headers – same User-Agent, same Accept, same everything – from every IP. That created a fingerprint that Google could detect across the pool. Once I randomised headers per request, the block rate dropped from 40% to 5%. I now maintain a list of 50 realistic User-Agent strings and rotate them.

Another mistake was ignoring JavaScript rendering. For many SERP features – like knowledge panels, local packs, and ads – Google serves the HTML only after JavaScript executes. My initial scraper used plain HTTP requests and missed half the page. I switched to a headless browser controlled by Puppeteer, which executes JS. That doubled the data I could extract from each scrape. But this increased load time per request from 0.4 seconds to 2 seconds, so I had to reduce throughput. I now use a hybrid approach: plain requests for simple result lists, headless browser for rich results.

I also underestimated the importance of session persistence. Some ranking tools maintain a session across multiple requests to simulate a user's browsing behaviour. When I rotated IPs too aggressively, I lost the session and got served different results. I now use sticky sessions – the same IP for the duration of a single keyword scrape – and only rotate between keywords. This fixed the inconsistency in my data.

Finally, I thought more proxies was always better. I bought a pool of 1,000 IPs and spread my 200 keywords across them. But the proxy provider's pool was shared, so many IPs were already blocked by Google. The effective pool was only about 200 usable IPs. I now test each batch of new proxies against a known clean query before adding them to my pool. I also limit the pool size to what I actually need – 200 IPs for 100 keywords per location – rather than buying in bulk.