1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Whats your wordpress robots.txt look like?

Discussion in 'White Hat SEO' started by mikie46, Jun 15, 2011.

  1. mikie46

    mikie46 Jr. VIP Jr. VIP

    Joined:
    Aug 6, 2008
    Messages:
    1,454
    Likes Received:
    1,102
    Iv spent a huge amount of time trying to come up with a best robots.txt file for wordpress but recently, due to ranking drops im wondering if my robots.txt is killing my rankings.

    Here is mine; What does your look like?

    Code:
    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/plugins/
    Disallow: /wp-content/cache/
    Disallow: /wp-content/themes/
    Disallow: /recommends/
    Disallow: /wp-login.php
    Disallow: /*wp-login.php*
    Disallow: /trackback/
    Disallow: */feed*
    Disallow: /comments/
    Disallow: /author/
    Disallow: /contact/
    Disallow: */trackback*
    Disallow: /archives/
    Disallow: */comments/
    Disallow: /page/
    Disallow: /tag/
    Disallow: /z/j/
    Disallow: /z/c/
    Disallow: /stats/
    Disallow: /dh_
    Disallow: /category/*
    Disallow: /category/
    Disallow: /login/
    Disallow: /20*
    Disallow: /iframes/
    Disallow: /i/
    Disallow: /f/
    Disallow: /t/
    Disallow: /c/
    Disallow: /j/
    Disallow: /*.php$
    Disallow: /*?*
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /*.gz$
    Disallow: /*.wmv$
    Disallow: /*.cgi$
    Disallow: /*.xhtml$
    Disallow: /*?*
    Disallow: /*?
    
    # alexa archiver
    User-agent: ia_archiver
    Disallow: /
    
    # disable duggmirror by Digg
    User-agent: duggmirror
    Disallow: /
    
    # allow google image bot to search all images
    User-agent: Googlebot-Image
    Disallow: /wp-includes/
    Allow: /wp-content/uploads/
    
    Sitemap: http://mydomain.com/sitemap.xml.gz
     
  2. mikie46

    mikie46 Jr. VIP Jr. VIP

    Joined:
    Aug 6, 2008
    Messages:
    1,454
    Likes Received:
    1,102
    Nobody uses it huh? Interesting.
     
  3. cnick79

    cnick79 Jr. VIP Jr. VIP

    Joined:
    Jun 10, 2010
    Messages:
    653
    Likes Received:
    341
    Location:
    Google's SandBox
    Code:
    User-agent: *
    Disallow: *
     
  4. SeRf*X

    SeRf*X Registered Member

    Joined:
    Apr 5, 2011
    Messages:
    59
    Likes Received:
    12