Duplicate Content: Why It Happens and How to Fix It

Duplicate content is a source of constant anxiety for many site owners.

Read almost anything about it, and you'll come away believing that your site is a ticking time bomb of duplicate content issues. A Google penalty is merely days away.

Thankfully, this isn't true—but duplicate content can still cause SEO issues. And with 25-30% of the web being duplicate content, it's useful to know how to avoid and fix such issues.

In this guide, you'll learn:

Duplicate content is the same or similar content that appears on the web in more than one place. It can exist on one website or across multiple websites.

For example, let's play a game of spot the difference...

Here's the page located at caltonnutrition.com/tag/protein-powder/...

... and here's the page located at caltonnutrition.com/tag/whey/:

Notice any difference? Me neither. The content at both URLs is identical. AKA duplicate content.

Google says there's no such thing as a duplicate content penalty. But duplicate content can hurt your SEO performance for a few reasons.

Imagine that the same page is available at three different URLs:

The first should show up in search results, but Google can get this wrong. If that happens, an undesirable URL may take its place.

Because people may be less inclined to click on an unfriendly URL, you may get less organic traffic.

If the same content is available at many URLs, then each of those URLs may attract backlinks. That results in the splitting of "link equity" between URLs.

To show an example of this in the wild, take a look at these two pages on buffer.com:

These pages are almost exact duplicates. And they have 106 and 144 referring domains (links from unique websites), respectively.

Before you panic, know that this isn't always a problem because of how Google handles duplicate content.

In simple terms, when they detect duplicate content, they group the URLs into one cluster. They then "select what [they] think is the 'best' URL to represent the cluster in search results" and "consolidate properties of the URLs in the cluster, such as link popularity, to the representative URL." This process is known as canonicalization.

So, in the case above, Google should show only one of the URLs in organic search and attribute all referring domains in the cluster (106+144) to that URL.

But that's not what happens, as we see both URLs ranking in Google for similar keywords.

In this instance, Google likely isn't consolidating "link equity" at one URL.

Google finds new content on your website via crawling, which means they follow links from existing pages to new pages. They also recrawl pages they know about from time to time to see if anything has changed.

Having duplicate content serves only to create more work for them. That can affect the speed and frequency at which they crawl your new or updated pages.

That's bad because it may lead to delays in indexing new pages and reindexing updated pages.

Occasionally, you may permit another website to republish your content. That's known as syndication. Other times, sites may scrape your content and republish it without permission.

Both of these scenarios lead to duplicate content across multiple domains, but they usually don't cause problems. It's only when the scraped or republished content starts outranking the original on your site that issues arise.

The good news is this is a rare occurrence, but it can happen.

Google has stated on multiple occasions that they don't have a duplicate content penalty.

But, this isn't entirely true. If your duplicate content is accidental and not the result of intentional manipulation of search results or spammy practices, then you won't get penalized. If it is, then you might.

Google confirms that here:

The question is, what counts as "intent to manipulate our rankings and deceive our users"?

Google has a lot of information on that here. But basically, it's things like:

However, as discussed above, duplicate content can still hurt SEO—even without a penalty.

There's no single cause of duplicate content. There are many.

Faceted navigation is where users can filter and sort items on the page. Ecommerce websites use it a lot.

This kind of navigation appends parameters to the end of the URL.

Because there are usually many combinations of these filters, faceted navigation often results in lots of duplicate-or-near-duplicate content.

Take a look at these two pages, for example:

The URLs are unique, but the content is almost identical.

Plus, the order of the parameters often doesn't matter. For example, the same page is accessible at both of these URLs:

Parameterized URLs are also used for tracking purposes. For example, you may use UTM parameters to track visits from a newsletter campaign in Google Analytics:

Example: example.com/page?utm_source=newsletter

Session IDs store information about your visitors. They usually append a long string to the URL like so:

Example: example.com?sessionId=jow8082345hnfn9234

Most websites are accessible at one of these four variations:

If you're using HTTPS, it'll be one of the first two. Whether it's the www or non-www version is your choice.

However, if you don't correctly configure your server, your site will be accessible at two or more of these variations. That isn't good and can lead to duplicate content issues.

Google sees URLs as case-sensitive.

That means these three URLs are all different:

Google treats URLs with and without trailing slashes as unique. That means these two URLs are unique in Google's eyes:

If your content is accessible at both URLs, then that can lead to duplicate content issues.

To check if this is an issue, try to load a page with and without the trailing slash. Ideally, only one version will load. The other will redirect.

For example, if you try to load this post without the trailing slash, it will redirect to the URL with the trailing slash.

Google states that this behavior is ideal.

Print-friendly versions have the same content as the original. It's only the URL that differs.

Mobile-friendly URLs, like print-friendly URLs, are duplicates.

Accelerated Mobile Pages (AMP) are duplicates.

Most CMS' create dedicated tags pages when you use tags.

For example, if you have an article about organic whey protein, and you use both "protein powder" and "whey" as tags, then you'll end up with two tag pages like these:

That doesn't always cause duplicate content in itself, but it can.

That's the case here because there's only one page on the site with those two tags—so each tag page is identical.

Many CMS' create dedicated pages for image attachments. These pages usually show nothing but the image and some boilerplate copy.

Because this copy is the same across all auto-generated pages, it leads to duplicate content.

WordPress and other CMS' allow for paginated comments. This causes duplicate content as it effectively creates multiple versions of the same URLs.

If you're serving similar content to people in different locales who speak the same language, then that can cause duplicate content.

For example, you might have different versions of your site for people in the US, UK, and Australia. Because there are likely only minor differences between the content served to each locale (e.g., prices in dollars versus pounds sterling), the versions will be near duplicates.

Lots of websites have search boxes. Using these typically takes you to a parameterized search URL.

Example: example.com?q=search-term

Google's former Head of Webspam, Matt Cutts, stated that:

A staging environment is a duplicate or near-duplicate version of your site used for testing purposes.

For example, imagine that you want to install a new plugin or change some code on your website. You might not want to push that straight to a live site with hundreds of thousands of daily visitors. The risk of catastrophe is too high. The solution is to test the changes in a staging environment first.

Staging environments become an SEO issue when Google indexes them because it results in duplicate content.

Head over to Ahrefs' Site Audit and start a crawl.

Once done, head to the Content quality report.

Look for clusters of duplicates and near-duplicates without a canonical. These are highlighted in orange.

Click any of these clusters to see the affected pages.

Investigate the reason for the duplicate content, then take the appropriate action.

Note that these won't always be issues that need rectifying, especially in the case of near duplicates.

You can also check for duplicate title tags, meta descriptions, and H1s in the HTML tags report.

Bad duplicates are what you're looking for. These are pages with duplicate meta tags but different canonicals.

Select these by clicking the "Bad duplicates" toggle under HTML tags & content.

Click on any of the yellow bars to see the affected pages.

Pages with duplicate titles, meta descriptions, or H1's are often very similar.

For example, these two have the same title tag, and the content is almost identical because the product is the same. The only difference is that one of the pages is for a 3-pack of instant lighting firelogs, whereas the other is for just one.

Google states that you should minimise similar content like this:

However, a small number of similar pages is unlikely to be much of an issue.

Content scraping and syndication can also lead to duplicate content issues. But it's only usually an issue if you see scraped versions of your content outranking you.

Does that happen? Yes, but it's often more of an issue for new or weak websites. Why? Because the sites scraping your content are often more authoritative. That sometimes "tricks" Google into thinking that theirs is the original.

If you have a small website, then you can often find scraped content by searching Google for a snippet of text from your page in quotes.

For larger sites, you'll need to use an automated tool like Copyscape. This searches the web for other occurrences of the content on your page(s).

Whichever method you use, most results will be from spammy and low-quality sites.

Generally speaking, these are nothing to worry about. However, if you see that a legitimate website scraped your content, and are concerned that it may be stealing your traffic, throw the URL into Ahrefs' Site Explorer to see an organic traffic estimate.

If it's getting more traffic than your page, then there may be an issue.

In this case, you have three options:

If you intentionally syndicate content to other websites, then it's worth asking them to add a canonical link to the original. That will eliminate the risk of duplicate content issues.

Don't stress over duplicate content too much. It's usually much less of an issue than it's thought to be.

If you have a handful of duplicate or near-duplicate pages, there's unlikely to be much of a problem. The same is true when quoting content from another website or other pages on your site. Small amounts of duplicate or boilerplate content should be okay. Google has systems in place to deal with such things.

What you need to be on the lookout for are technical SEO mishaps that lead to the generation of hundreds or thousands of pages of duplicate content, such as the improper implementation of faceted navigation on ecommerce sites.

These can wreak havoc on your crawl budget, amongst other things.

Let me know in the comments or on Twitter if you're struggling with duplicate content.