1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

[HELP] Should I disallow tag/category/archives/feeds in robot.txt?

Discussion in 'Black Hat SEO' started by SuperNoobInc, May 29, 2012.

  1. SuperNoobInc

    SuperNoobInc Regular Member

    Joined:
    Dec 28, 2011
    Messages:
    467
    Likes Received:
    32
    Do you guys disallow tags, feeds, categories, archives in your robot.txt?


    When I type site:mydomain.com
    I was surprise to see XXX pages indexed, while I technically only have around 30 readable-pages.
    All others are tags/categories/archives etc. Do they come in handy in SERP?


    Also, should I bother disallowing wp-admin and wp-include from crawlers?


    Thanks guys.
     
  2. SuperNoobInc

    SuperNoobInc Regular Member

    Joined:
    Dec 28, 2011
    Messages:
    467
    Likes Received:
    32
    Some help with this will be great. Thank you
     
  3. Seo Lover

    Seo Lover Jr. Executive VIP Jr. VIP Premium Member

    Joined:
    Jan 30, 2011
    Messages:
    5,696
    Likes Received:
    4,126
    Gender:
    Male
    Occupation:
    Hanging Around Interwebs !
    Location:
    <-----------------Sin City
    Allow it , more pages indexed more SEO friendly site

    So dont disallow it
     
    • Thanks Thanks x 2
  4. healzer

    healzer Jr. Executive VIP Jr. VIP Premium Member

    Joined:
    Jun 26, 2011
    Messages:
    2,364
    Likes Received:
    1,967
    Gender:
    Male
    Occupation:
    Marketing automation tools
    Location:
    Somewhere in Europe
    Home Page:
    It is important you do not include "secret" pages in your robots.txt
    especially like admin log in and so on, because anyone is able to view your robots.txt file

    just a quick tip ;)
     
    • Thanks Thanks x 1
  5. SuperNoobInc

    SuperNoobInc Regular Member

    Joined:
    Dec 28, 2011
    Messages:
    467
    Likes Received:
    32
    So for a typical WP site, what should be 'disallowed' in general?
     
  6. studentcashnow

    studentcashnow Regular Member

    Joined:
    Aug 21, 2009
    Messages:
    246
    Likes Received:
    186
    you can rank tags and get more traffic, rank categories get more traffic, which could help rank the pages in specific categories by passing link juice
     
  7. ryanhackmachine

    ryanhackmachine Regular Member

    Joined:
    May 8, 2010
    Messages:
    380
    Likes Received:
    223
    Occupation:
    Spend Money
    Location:
    In front of my desktop pc
    ive been wondering about this too! good thing to know i didnt disallowed them
     
  8. SuperNoobInc

    SuperNoobInc Regular Member

    Joined:
    Dec 28, 2011
    Messages:
    467
    Likes Received:
    32
    So far, none of my tags / categories are ranking well.
    However, when I type site:mydomain.com

    I do see all my indexed pages, including tags/categories and others.
    However, there are NO title nor metadescription for those tags/categories pages
    They simply shown as links (url) on search results...