1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Robots.txt for tags

Discussion in 'Black Hat SEO' started by Whistleblower, Dec 14, 2015.

  1. Whistleblower

    Whistleblower Junior Member

    Joined:
    Nov 5, 2015
    Messages:
    135
    Likes Received:
    33
    is this format correct for this ?
    User-agent: *
    Disallow: /wp-admin/
    Disallow: /tag/abcd
    Disallow: /tag/efgh
    Allow: /wp-admin/admin-ajax.php

    Do you simply do disallow or do something else as well along with disallow ?

    Thanks in advance.
     
  2. Whistleblower

    Whistleblower Junior Member

    Joined:
    Nov 5, 2015
    Messages:
    135
    Likes Received:
    33
    Anyone. i need confirmation. anyone ?
     
  3. Resolution10

    Resolution10 Power Member

    Joined:
    Jan 6, 2012
    Messages:
    718
    Likes Received:
    295
    That's correct in general, but what exactly are you trying to do with those tags? You may consider wildcard matching for these, or simply go for a plugin like Yoast to noindex all tags, unless you are using them as posts or pages of course. Or go with no-archive and no-snippet X-robots to ensure they won't get picked either way.

    Also, make sure you are not blocking other required scripts in wp-admin.
     
  4. Whistleblower

    Whistleblower Junior Member

    Joined:
    Nov 5, 2015
    Messages:
    135
    Likes Received:
    33
    Those "abcd" and "efgh" are just examples. I want to no index some selected ones, is that correct and i don't know about wildcard. whats that ? (i have check its meaning first) can you explain it a bit or link me to a better info source ?

    thank u
     
  5. Resolution10

    Resolution10 Power Member

    Joined:
    Jan 6, 2012
    Messages:
    718
    Likes Received:
    295
    WP tags can be quite evil in terms of duplicate content. In rare occasions or bigger websites tags might be used as actual pages on the website. In that case they need to stay indexed obviously.

    But more often than not, you would like all of them deindexed. They serve their purpose from an UX perspective but there's absolutely no reason to have them indexed. If you don't want to go with Yoast which handles them with a tick-box click, wildcard would match all your tags with "Disallow: /*?" which would pick everything starting with "?" such a ?tag, ?attachment_id etc.

    If you still want to just go with individual tags just follow your initial robots structure in the OP.
     
    • Thanks Thanks x 1