SEO Testing: A Simple (But Complete) Guide

But when should you do SEO testing, and how?

In this guide, you’ll learn:

• What SEO testing is
• Why you should run SEO tests
• Types of SEO tests
• If SEO testing is right for you
• How to run an SEO test
• Examples of SEO tests

SEO testing is the process of making changes to your website and measuring the impact on organic search traffic or rankings.

Here are four benefits of running SEO tests:

1. See what works for your website

Known ranking factors aside, there’s a lot of “it depends” scenarios in SEO. Just because someone saw a positive impact after making a change doesn’t mean things will play out the same way for you. It’s better to test it on your website.

2. Justify resource allocation

Rolling out SEO changes can absorb a lot of time and resources. If that change doesn’t have the desired effect, it’s a waste of energy. Testing tells you whether something is likely to have a positive impact in less time using fewer resources.

3. Be data-driven, not opinion-driven

Everyone has an opinion on what does and doesn’t work in SEO. Often the quickest way to settle disputes is to test them. Fail to do this, and the person with the loudest voice tends to dictate direction—which isn’t always for the best.

4. Make sure changes won’t backfire

If there’s one thing worse than spending time and effort rolling out an SEO change that has no impact on SEO, it’s doing the same for something that negatively impacts SEO. By running an SEO test, you can limit any negative outcome to a small subset of pages.

There are three main types of SEO tests.

1. SEO serial testing (not recommended)

Serial testing is where you change all of the pages on your website at once and observe the outcome. We wouldn’t recommend this for three reasons:

1. If the change negatively affects SEO, it impacts all your pages—not just a few.
2. It’s usually more time-consuming to make a sitewide change and even more time-consuming if you have to roll it back.
3. It doesn’t account for seasonality or variables outside of your control.

2. SEO time-based testing (not recommended)

Time-based testing is where you make a change to one page and see how it performs. We don’t recommend this type of SEO testing because it’s unwise to assume causation with a sample size of one.

3. SEO A/B testing

A/B testing (or split-testing) is where you change some pages but not others. You then compare how the changed pages fare against the unchanged pages. The group of unchanged pages is known as your control group, and the group of changed pages is known as the variant group.  We recommend this type of SEO testing because:

1. If the change negatively affects SEO, it impacts only a small subset of pages.
2. It’s not as time-consuming as serial testing because you only have to make changes to a small subset of pages.
3. It helps account for seasonality and variables outside of your control because these things affect both groups of pages.

SEO testing probably isn’t right for you unless your website gets a significant amount of organic traffic. We’re talking tens or hundreds of thousands of organic visits per month. There are two reasons for this:

1. There are better ways to spend your time. If your website doesn’t get much traffic, SEO testing isn’t going to solve this. Instead, focus on creating more search-focused content, building more backlinks, and other techniques known to improve SEO.
2. Results will be statistically insignificant. If you asked three people to name their favorite band, and they all said BTS, you wouldn’t assume everyone else felt the same. You probably just chanced upon some die-hard BTS fans. However, if you asked 10,000 people, and they all said BTS, it’d be fair to assume most people are fans. It’s the same story with SEO testing. It’s unwise to assume causation between a change and observed outcome without a substantial sample.

If your website gets plenty of traffic, improving rankings or increasing traffic by a small percentage can significantly impact your business—and this is where SEO testing makes sense.

Follow these seven steps to get started with SEO testing:

1. Form a hypothesis

A hypothesis is a prediction. It’s where you decide what you’re going to change and how you think it’ll affect your website’s SEO visibility.

Here’s a simple formula for forming a hypothesis:

[change] will lead to [effect] on [types of pages]

Here’s an example:

[adding short descriptions] will lead to [a 10% improvement in organic traffic] on [patio furniture ecommerce category pages]

You can see that we’ve outlined what we’ll change, the intended outcome, and the types of pages we’ll use for the test.

Make sure to choose a group of pages that have a lot in common when doing this—like blog posts or e-commerce category pages—to minimize variables that can screw up your test results. Your hypothesis should also be an educated guess, not just something random.

2. Choose your pages

It’s best practice to only run tests with pages that get a decent amount of organic traffic. This is because you’re not going to learn much from a page that gets little or no organic traffic. It’ll just skew your test results.

You can find pages with decent organic traffic in Google Analytics:

1. Go to Behavior > Site Content > Landing pages
2. Filter by organic traffic only
3. Filter for the types of pages you want to test (e.g., pages with /blog/ in the URL)
4. Filter for pages with at least [x] sessions

If you don’t use Google Analytics, you can do the same thing using Ahrefs’ Site Explorer.

1. Enter your domain into Site Explorer
2. Go to the Top Pages 2.0 report
3. Filter for pages that get at least [x] amount of traffic per month
4. Filter for the kinds of pages you’re interested in based on footprints (e.g., /blog/ in the URL) using the Include filter

Download the results and clean them up where required. These are the pages that you’ll use for your test.

3. Pull a random sample

You’ll be running an AB split-test, so you’ll only need to make changes to a random subset of pages. The remaining pages will form your control group.

If you’re using a dedicated SEO testing tool, it may handle random sampling for you. However, if you’re doing things manually, the easiest way to pick a random sample is using the “Randomize range” function in Google Sheets.

Here’s how to do it:

1. Important your dataset (URLs, etc.) into Google Sheets
2. Highlight all the data
3. Right-click and hit “Randomize range”

This randomizes the order of URLs, meaning you can now use a percentage of them from the top as your variant group.

4. Decide on the test duration

Knowing how long to run a test for can be difficult, but it’s basically as long as it takes to gather a decent amount of data. This could be anywhere from a few days to a few months. It depends on how much traffic you get and how many pages you have.

Here are a couple of pointers:

• Run the test long enough for Google to re-crawl the pages in your variant group. You can’t attribute any fluctuations in traffic or rankings to your changes unless the pages have been re-crawled.
• Run the test long enough to get a decent amount of traffic to your pages. Don’t make any conclusions until your pages have received significant traffic, as they’ll likely be statistically insignificant.

If you’re not sure how long to go for, I recommend starting with a month. You can always leave the test running for longer if you need more data.

5. Decide how to track test results

Before you make changes to your site, you need a way to track results. The easiest way to do this is to use an SEO testing tool. It connects to your Google Search Console account and tracks results for you.

But you can track results manually if you prefer.

If you want to test changes in organic traffic, use Google Search Console or Google Analytics. Just compare average differences in traffic to both groups for the period before and after the changes were made.

If you want to test changes in CTR, use Google Search Console and compare average changes.

If you want to track changes in organic rankings, use Google Search Console or a rank tracking tool like Ahrefs’ Rank Tracker. The latter is arguably the better option because it gives you actual ranking positions—not just averages.

To set this up in Ahrefs’ Rank Tracker, add and tag two batches of keywords:

1. Control group keywords tagged as “control”
2. Variant group keywords tagged them as “variant”

You can then see average ranking positions for the keywords in each group in the Tags report:

6. Make the changes

It’s finally time to implement your AB test. Here are a few pointers for doing this:

• Don’t make changes to pages in the control group. It’s called the control group for a reason; it’s your baseline against which you’ll evaluate the impact of your change.
• Change one variable only. If you change multiple things, you’ll have trouble figuring out whether it was one or a combination of things that led to the outcome.
• Keep a record of the changes you made and the date you made them.
• Make sure you have a way to revert back. You’ll need to do this if your SEO test produces an inconclusive or negative outcome.
• Don’t test pointless things. If you can quickly and easily implement an SEO best practice, don’t bother testing it—just do it.

7. Check the results

If you’re using a testing tool, it’ll probably produce some nice graphs for you to analyze once the test is over.

If not, you can evaluate the outcome of your test manually using Google Analytics (if you’re testing for changes in traffic), Google Search Console (if you’re testing for changes in traffic, rankings, or CTR), or a rank tracking tool (if you’re testing for changes in rankings).

If you’re testing for changes in rankings and everything’s set up in Ahrefs’ Rank Tracker, as shown in step five, you can see changes in tracked keyword visibility since the start of your test in the Tags report. Just choose the timeframe.

Everyone should run different SEO tests depending on their SEO goals. But let’s look at a few examples of common tests to drill the process home.

1. Testing the impact of title tag changes on CTR for e-commerce product or category pages

As title tags show up in the search results, they can affect your pages’ clickthrough rates (CTR). If you use the same formula for all product or category pages like most ecommerce sites, testing the impact of a different formula on CTR is pretty straightforward.

How to test

Take a bunch of e-commerce category pages that get a decent amount of organic traffic, take a random sample, then change the title tag formula on those pages.

For example, let’s say that your current formula for product pages is:

[Product name] | [Brand]

You might want to test whether adding “Free delivery” improves CTR:

[Product name] - Free Delivery | [Brand]

How to measure

Use Google Search Console to compare the average CTR for pages in both groups for the period before and after the changes.

2. Testing the impact of a content optimization tool on blog post rankings

Many SEO professionals swear by content optimization tools—but many others don’t. If you’re in the camp that thinks there might be something to this approach, why not test it?

How to test

Take a bunch of blog posts that get a decent amount of organic traffic and rank in the top 10 for their main target keywords, take a random sample, then “optimize” those posts with a content optimization tool.

Here’s how to find such blog posts in Ahrefs’ Rank Tracker:

1. Choose your project
2. Go to the Pages report
3. Filter for top 10 rankings
4. Filter for pages with at least [x] monthly estimated organic traffic from tracked keywords
5. Export the results

How to measure

Use Ahrefs’ Rank Tracker to compare the average visibility for tracked keywords in each group on the test start and end dates as explained in step five.

3. Testing the impact of adding snippet-friendly content on blog post featured snippet rankings

Ranking in featured snippets usually drives more clicks. So if you have blog posts targeting and ranking for keywords that trigger featured snippets, why not test your ability to win them?

How to test

Take a bunch of blog posts that get a decent amount of organic traffic and rank in the top 10 for keywords that trigger featured snippets, take a random sample, then add snippet-friendly content to those posts.

Here’s how to find such posts in Ahrefs’ Site Explorer:

1. Enter your blog/website into Site Explorer
2. Go to the Organic Keywords 2.0 report
3. Filter for top 10 rankings
4. Filter for keywords that send at least [x] monthly estimated organic traffic (so that you know the pages get a decent amount of traffic)
5. Filter for unowned featured snippets
6. Export the results

How to measure

Use Ahrefs’ Rank Tracker to compare the number of owned featured snippets for tracked keywords in each group on the test start and end dates.

Final thoughts

To keep this article as simple as possible, I intentionally avoided the issue of statistical significance. This can get real complicated real quick, but it’s basically a way to prove that the results of an SEO test are reliable and that there’s a high probability of a causal relationship between the change and outcome. In other words—that whatever happened (or didn’t happen) is unlikely to have been by chance.

I’m not currently aware of any SEO testing tool that calculates this for you. So let me reiterate my earlier advice: don’t run SEO tests unless your website gets plenty of traffic.

If you’re running tests and making conclusions after only a few hundred visits, chances are the results aren’t going to be statistically significant.

Got questions? Ping me on Twitter.