Duplicate pages

Why one page ends up available at different URLs, when that hurts search and how to choose the main address.

A product page opens at two URLs. A visitor will probably notice nothing, but a search engine has to decide which address to show people. Pages with identical or very similar content like this are called duplicates.

A couple of repeating URLs does not mean the site has been penalised. The problem starts when extra versions keep appearing: search engines spend time crawling them, an inconvenient URL may end up in results and the team finds it harder to see which pages actually work. The fix starts with choosing the page you want to keep as the main one.

If you don't choose the main version, search will

Imagine one service opens at /service and at /service/. For the company it is one page, but for a search engine these are two different addresses. It may merge them and pick the one it considers main for its results, which will not necessarily match the URL in your menu or ads.

The fact that one duplicate is excluded from search is not always an error. The search engine may already have made the right choice. What needs fixing is a situation where it shows the wrong version, crawls lots of useless URLs or skips pages you were counting on.

Sometimes the site produces extra URLs itself

Duplicates are rarely created by hand. They come from the rules the site uses to build links:

• Two paths to one product. A product page opens from different catalogue sections at different URLs, though the content is the same.
• Parameters in links. Ad tags or sorting settings are appended to the URL. The page looks almost the same, but the link is different.
• URL variants. The site opens a page both with and without a trailing slash, with and without www, or at an old and a new address.
• CMS settings. One page is available both at a readable URL and at a technical link with a record ID.

Not every similarity is a duplicate. Two product pages can have similar descriptions but represent different products. And city pages may be needed separately if their terms and content differ. Merging them just for a cleaner report is a mistake.

First find two pages competing for one place

Start with specific URLs. Open both pages and compare not just the title but the main text, the product or service and the purchase terms. Then check which URL appears in the menu and internal links, and which one the search engine treats as main.

Google Search Console reports help you see pages that search has recognised as duplicates or excluded from results. But a status alone, without checking the pair of URLs, says little. Sometimes exclusion is correct: the right version is already in search. If the versions really are redundant, find the rule that creates them, otherwise new ones will appear after you fix a few links.

Redirect an unneeded URL to the main one

When a separate page at the second URL is no longer needed, set up a permanent redirect to the main one. People following an old link land on the working page, and search gets a clear signal about which address to prefer. This is the usual approach after renaming a page or choosing one variant, with or without a trailing slash.

Check the menu, in-text links and sitemap at the same time: they should point to the chosen URL too. Otherwise the site keeps offering old variants to search even when the redirect works. When migrating a live site, this belongs in the pre-launch check we describe in the article on SEO support.

If both URLs are needed, state the preferred one

Sometimes an alternative URL cannot simply be removed. For example, a visitor changes the order of products in the catalogue and the link gets a sorting parameter. When the content stays the same or nearly the same, the developer can specify the main URL in the page code with rel="canonical".

This is a hint to the search engine, not an order. If the pages differ noticeably or the main URL is unavailable, it may choose differently. So adding the setting to the code is not enough: check whether the search engine's choice matches yours.

Not every extra page should be redirected

Internal search results and some filter variants can be useful to visitors but unnecessary in search results. If such a page should open for people but stay out of search, the developer can add noindex. The crawler must be able to access the page, otherwise it will not see this directive.

The robots.txt file does something else: it limits crawling of URLs. Blocking crawling alone does not guarantee that a URL disappears from search results. A 404 response is appropriate when a page really no longer exists and there is no suitable replacement. Do not send a live product or service into nowhere just to clean up a report.

After the fix, check the result, not just the setting

Open the old URL and make sure it leads where it should. Check the main URL: it must work and be accessible to search. If you used a canonical hint or noindex, make sure it is on the right pages and not accidentally on a whole section.

Changes in search engine reports appear after recrawling, not at the moment you publish the fix. For the client, the result is simple: the important page is available at the expected URL, extra versions do not push it out of results, and new duplicates stop appearing.

Don't start by mass-closing URLs. Take a few specific pairs, decide which page people and search need, and find the reason the second version appears. Then the developer can choose the right rule for the whole site.