1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

what about this robots.txt for wordpress?

Discussion in 'Black Hat SEO' started by iamok22, Feb 4, 2010.

  1. iamok22

    iamok22 Regular Member

    Joined:
    Dec 24, 2009
    Messages:
    295
    Likes Received:
    4
    is it good in your opinion?

    User-agent: *
    # disallow all files in these directories
    Disallow: /cgi-bin/
    Disallow: /z/j/
    Disallow: /z/c/
    Disallow: /stats/
    Disallow: /dh_
    Disallow: /about/
    Disallow: /contact/
    Disallow: /tag/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /contact
    Disallow: /manual
    Disallow: /manual/*
    Disallow: /phpmanual/
    Disallow: /category/
    User-agent: Googlebot
    # disallow all files ending with these extensions
    Disallow: /*.php$
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /*.gz$
    Disallow: /*.wmv$
    Disallow: /*.cgi$
    Disallow: /*.xhtml$
    # disallow all files with ? in url
    Disallow: /*?*
    # disable duggmirror
    User-agent: duggmirror
    Disallow: /
    # allow google image bot to search all images
    User-agent: Googlebot-Image
    Disallow:
    Allow: /*
    # allow adsense bot on entire site
    User-agent: Mediapartners-Google*
    Disallow:
    Allow: /*

    Sitemap: http://www.**********.com/sitemap.xml
     
  2. richcamp

    richcamp Regular Member

    Joined:
    Oct 5, 2009
    Messages:
    315
    Likes Received:
    119
    I dont think dissalowing tags, category, about and contact are good for WP. Tags and category are good for relevancy, about and contact are good for credibility.

    Plugins is the one directory you dont want bots to snoop around and even that only if you have some shady plugins
     
  3. iamok22

    iamok22 Regular Member

    Joined:
    Dec 24, 2009
    Messages:
    295
    Likes Received:
    4
    about tags: it is made in that way to avoid duplicate content...anyway i am not sure about this.some pople say that i should use tags and some not.

    about Disallow: /category/ it is made to avoid duplicate content and since i give a specific name to each category i think it won't be a problem.
    this will only prevent the bot to access the "/category/" not gategories with a specific name assigned.

    let me know if i am wrong...
     
  4. AltaVega

    AltaVega Newbie

    Joined:
    Dec 23, 2009
    Messages:
    23
    Likes Received:
    8
    you are right about "category" and "tag" permalinks creating duplicate content; but, I think you best use some sort of plugin to manage that problem,and not robots.txt.
    sure,best use robots.txt to "hide" some of the directories but think; if for any reason you change the permalink structure you'll have to change robots.txt, etc, meanwhile the plugin takes care by itself about those problems.