URL Parameters And Duplicate Content In Search Console


Block crawling of parameterized duplicate content, When and how to use the URL Parameters tool





URL parameters and duplicate content





If your site uses URL parameters for insignificant page variations (for instance, color=red vs color=green), or if your site uses parameters that can show essentially the same content using different URLs (for example, example.com/shirts?style=polo,long-sleeve and example.com/shirts?style=polo&style=long-sleeve), Google might be crawling your site inefficiently.





Here is an example of URLs that lead to essentially duplicate content, distinguished only by different parameters:





URLDescription
https://example.com/products/women/dresses/green.htmlStatic, non-parameterized page
https://example.com/products/women?category=dresses&color=greenURL uses  parameters category and color to deliver the same content as non-parameterized page.
https://example.com/products/women/dresses/green.html?limit=20&sessionid=123URL includes parameters to limit the number of results,  and a session ID for the user to show the same content.




If you have many such URL parameters in your site, then you might benefit by using the URL Parameters tool to reduce crawling of duplicate URLs. Important: If your site serves duplicate content to different URLs without using parameters, you should define a canonical page rather than block crawling, as described in this page.









Block crawling of URLs containing specific parameters





You can prevent Google from crawling URLs that contain specific parameters, or parameters with specific values, in order to avoid crawling duplicate pages.





Requirements





You should use the URL Parameters tool only if your site fulfills ALL of the following requirements.





  • Your site has more than 1,000 pages, AND
  • In your logs, you see a significant number of duplicate pages being indexed by Googlebot, in which all duplicate pages vary only by URL parameters (for example: example.com?product=green_dress and example.com?type=dress&color=green).




Incorrect usage warning You should use the URL Parameters tool only if your site fulfills the requirements above, and you are an experienced SEO. Using the URL Parameters tool incorrectly can cause Google to ignore important pages on your site, with no warning or reporting about ignored pages. If this sounds a little dire, it's because many people misuse the tool, or use it unnecessarily. If you are unsure whether you are using this tool correctly, it's better not to use it.





Usage





You can specify Google's behavior when crawling your site with specific parameters. Parameter behavior applies to the entire property; you cannot limit crawling behavior for a given parameter to a specific URL or branch of your site.





To use the URL Parameters tool:





  1. Verify that your site meets the requirements listed previously.
  2. Open the URL Parameters tool.
  3. Either Edit an existing parameter, or click Add parameter to create a new one. Note that this tool is case-sensitive, so type your parameter name exactly as it appears in your URL.
  4. Specify whether your URL parameter affects page content:
  5. No: Doesn't affect page content: Your parameter doesn't affect how page content is presented. This type of parameter might be used to track visits and referrers, but has no effect on the actual content of the page. For example, sessionID or userName. If Google finds many URLs that differ only in this parameter value, it will crawl one of them. Google tries to detect these types of parameters, but if your logs indicate that we are not identifying this static parameter correctly, you can specify it here.
  6. Yes: Changes, reorders, or narrows page content:  Your parameter can change page content. Examples might be brandgendercountry, or sortorder. Choose the purpose of the parameter:
  7. Sorts (for example, sort=price_ascending): Changes the order in which content is presented.
  8. Narrows (for example, t-shirt_size=XS): Filters the content on the page.
  9. Specifies (for example, store=women): Determines the general class of content displayed on a page. If this specifies an exact item, and this is the only way to reach this content, should select "Every URL" for the behavior.
  10. Translates (for example, lang=fr): Displays a translated version of the content. If you use a parameter to show different languages, you probably do want Google to crawl the translated versions using hreflang to indicate language variants of your page rather than blocking content with this tool.
  11. Paginates (for example, page=2): Displays a specific page of a long listing or article.
     
  12. Which URLs with this parameter should Googlebot crawl? Choose an option to indicate Google's behavior when encountering URLs that contain this parameter:
  13. Let Googlebot decide: This setting is the default for already-known parameters. Select if you're unsure of a parameter's behavior, or if the parameter behavior changes for different parts of the site. Googlebot can analyze your site to determine how best to handle the parameter.
  14. Every URL: Tells Google never to block URLs with this parameter. URLs with unique values of this parameter do not contain duplicate content. For example, after you implement this type of setting for URLs containing the productid parameter, Google automatically considers the URL http://www.example.com/dresses/real.htm?productid=1202938 to be entirely different from http://www.example.com/dresses/real.htm?productid=5853729 because each URL has a different productid parameter value.
  15. Only URLs with value: Tells Google to crawl only URLs where your URL parameter is set to a specified value. URLs with a different parameter value won’t be crawled. This is particularly useful if your site uses the parameter value to change the order in which otherwise identical content is displayed. For example, http://www.example.com/dresses/real.htm?sort=price_high contains the same content as http://www.example.com/dresses/real.htm?sort=price_low. You could use this setting to tell Googlebot to crawl only those URLs where sort=price_low to avoid crawling the duplicate content.
  16. No URLs: Tells Google not to crawl any URLs with a specific parameter. Google won't crawl any URLs containing the parameter you entered. For example, you can tell Google not to crawl URLs with parameters such as pricefrom and priceto (like http://www.examples.com/search?category=shoe&brand=nike&color=red&size=5&pricefrom=10&priceto=1000) to prevent unnecessary crawling of duplicated content already available from http://www.examples.com/search?category=shoe&brand=nike&color=red&size=5.
  17. If your site uses multiple parameters in a URL, see managing URLs with multiple parameters.
  18. Note that your rules might be inherited by other properties (see Inheritance of parameter rules).




Inheritance of parameter rules





If you have separate properties for http and https, or separate parent and child properties (for example, example.com and example.com/fr/, or example.com and m.example.com) then your parameter settings might be inherited between properties, according to these rules:





  • http/https: If only one of your http or https properties has rules, then the rules are applied to both. If both your http and https properties have their own rules defined, then only their own rules are applied.
  • Parent/child: If a parent property (example.com) has parameter rules, any child property (example.com/fr/) without parameter rules inherits those rules; any child property with parameter rules uses only its own rules. Note that subdomains (m.example.com) count as children of parent domains (example.com).




Managing URLs with multiple parameters





A single URL can contain many parameters; you can specify crawling settings for each one separately. If a single URL contains multiple managed parameters, Google will obey the following rule when deciding whether to crawl the URL:





The more restrictive parameter settings override the less restrictive parameter settings.





For example, below are three URL parameters and their respective Google crawling settings:





ParameterParameter crawl settings
shopping-categoryCrawl all URLs with this parameter
sort-byCrawl only URLs with value = production-year
sort-orderCrawl only URLs with value = asc




Example 1





http://www.example.com?shopping-category=shoes&sort-by=size&sort-order=asc.





Google won't crawl this URL because the sort-by parameter is not set to production-year even though the URL contains a valid sort-order value (asc)





Example 2





http://www.example.com?shopping-category=DVD-movies&sort-by=production-year&sort-order=asc.





Google can crawl this URL because the sort-by and sort-order values match the allowed settings.





Example 3





http://www.example.com/shoes/33453





http://www.example.com?country=fr





Google can crawl both URLs because they don't have any flagged parameters.


Post a Comment

12 Comments

  1. I believe other website owners should take this website as an example , very clean and wonderful user friendly design and style.

    ReplyDelete
  2. Hi Dear, are you really visiting this web page daily, if so then you will absolutely get fastidious experience.

    asmr 0mniartist

    ReplyDelete
  3. Simply desire to say your article is as surprising.
    The clarity for your post is just cool and that i could suppose
    you are an expert on this subject. Well with your permission allow me to seize your feed to stay
    updated with approaching post. Thank you one million and please keep up the rewarding work.
    0mniartist asmr

    ReplyDelete
  4. I know this site presents quality dependent posts and extra information, is there any other site which offers
    such things in quality? asmr 0mniartist

    ReplyDelete
  5. Thank you, I have recently been looking for information approximately this subject for a long time and yours
    is the greatest I have discovered so far. But, what in regards to the bottom line?
    Are you positive concerning the supply? asmr 0mniartist

    ReplyDelete
  6. Hey! This post couldn't be written any better!
    Reading this post reminds me of my good old room mate!
    He always kept talking about this. I will forward this article to him.
    Pretty sure he will have a good read. Many thanks for sharing!

    ReplyDelete
  7. You actually make it seem so easy with your presentation but I find this topic to be actually something which I
    think I would never understand. It seems too complex and very broad for me.
    I'm looking forward for your next post, I'll try to get the hang of
    it!

    ReplyDelete
  8. Hello There. I found your blog the use of msn. That is an extremely neatly written article.
    I'll be sure to bookmark it and return to learn more
    of your useful info. Thanks for the post.
    I will definitely comeback.

    ReplyDelete
  9. Its not my first time to visit this web site, i am browsing this web page dailly and take
    good facts from here daily.

    ReplyDelete
  10. Peculiar article, totally what I needed.

    ReplyDelete
  11. My family members always say that I am wasting my time here at web,
    except I know I am getting knowledge all the time by reading thes pleasant articles or
    reviews.

    ReplyDelete
  12. Hi friends, how is everything, and what you would like to say regarding this paragraph, in my view its in fact remarkable in favor of
    me.

    ReplyDelete