SEO Ab Testing

I start by writing down one thing I want to change and why. For example, swapping the H1 on a set of category pages from "Buy [Product]" to "[Product] Reviews" because I think the latter matches the search intent better. I pick a template where the same change can be applied to many URLs — product pages, blog posts, location pages, anything that repeats. I aim for at least 20 pages per group, but I've gone as low as 12 when the site was small. I randomise the assignment: I sort the pages by URL alphabetically, then every other page goes to the control or variant group. I make the change on the variant group — only one variable, never two. I set up a 302 redirect from the original URL to the variant if I'm using a separate URL for the test, or I use a rel="canonical" on the alternate to point to the preferred version. I wait. I check Google Search Console's URL inspection tool to see if the variant pages have been indexed. I give it a minimum of two weeks, often three or four. Then I pull data from [Google Analytics](/google-analytics/) and [Looker Studio](/looker-studio/) for organic sessions and from Search Console for impressions and clicks. I compare the per-page averages between control and variant. I use a simple chi-squared test for click-through rate differences. If the variant shows a 10% or more lift in clicks with a p-value below 0.05, I consider rolling it out. If the data is inconclusive, I revert the change and move on. I do not report the test as a win unless I can replicate the result on a different page set later. This path has saved me from chasing ranking noise more than once.

The biggest watch-out is cloaking. Google's guidance is clear: do not show one version to Googlebot and another to users. If you use a separate URL for the variant, always add a rel="canonical" pointing to the control or use a 302 redirect. Never use a 301 unless you intend the change permanently. The second watch-out is sample size. I once ran a test on 5 pages. The variant showed a 30% lift in clicks. I was ready to celebrate. Then I realised the control group had two pages that dropped in rankings for unrelated reasons. The lift was not real. I now use a minimum of 10 pages per group, and I check that the groups have similar baseline performance before the test. The third watch-out is indexation lag. Googlebot may not reindex your variant pages for days or weeks. If you measure rankings the day after the change, you are measuring nothing. I wait until the variant URLs show as "crawled" in Search Console, then I add another week. The fourth watch-out is changing more than one variable. If you change the H1 and the meta description at the same time, you won't know which one caused the change. I keep it to one variable per test. The fifth watch-out is seasonality. A spike in clicks during a holiday week can look like a win. I compare the control group's performance to the variant's, not the variant's performance to itself last week. I also use [Google Tag Manager](/gtm/) to track events on the variant pages to ensure the user behaviour is real, not a bot or a fluke. Finally, I avoid testing on pages with very low traffic — fewer than 100 impressions per week per page. The noise is too high.

I got sample size wrong the first time. I tested a meta description change on 6 product pages. The variant won by 20% in clicks. I rolled it out to all 200 pages. Two weeks later, organic traffic to those pages dropped by 15%. The original test was a fluke — the control group had a few pages that were temporarily suppressed. I had to revert everything. I learned that small sample sizes produce false positives. I also got the timing wrong. I used to measure results after one week, not accounting for reindexing. I thought I had a winner, but Googlebot had not even visited the variant pages. The data was just pre-existing traffic. I now wait until I see the variant URLs in the Search Console performance report. Another thing I got wrong was testing multiple variables at once. I once changed the title, H1, and body copy on a set of blog posts. The variant won. I could not tell you which change helped. When I tried to replicate the win on a different set, I failed. I now change one thing per test. I also ignored the need for a proper tracking setup. I used raw Google Search Console data without filtering out branded queries. The variant looked great because it captured more brand searches. I now filter out branded queries in [SEO Analytics](/seo-analytics/) before analysing. And I once accidentally showed a different page to Googlebot than to users because I used a JavaScript redirect that Googlebot did not execute. That got me a manual action. I had to clean it up and submit a reconsideration request. That mistake cost me two months of rankings. I now test all redirects and canonical tags with the URL inspection tool before I start the test.