Whats your wordpress robots.txt look like?

Discussion in 'White Hat SEO' started by mikie46, Jun 15, 2011.

  1. mikie46

    mikie46 Jr. VIP Jr. VIP

    Joined:
    Aug 6, 2008
    Messages:
    1,454
    Likes Received:
    1,103
    Iv spent a huge amount of time trying to come up with a best robots.txt file for wordpress but recently, due to ranking drops im wondering if my robots.txt is killing my rankings.

    Here is mine; What does your look like?

    Code:
    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/plugins/
    Disallow: /wp-content/cache/
    Disallow: /wp-content/themes/
    Disallow: /recommends/
    Disallow: /wp-login.php
    Disallow: /*wp-login.php*
    Disallow: /trackback/
    Disallow: */feed*
    Disallow: /comments/
    Disallow: /author/
    Disallow: /contact/
    Disallow: */trackback*
    Disallow: /archives/
    Disallow: */comments/
    Disallow: /page/
    Disallow: /tag/
    Disallow: /z/j/
    Disallow: /z/c/
    Disallow: /stats/
    Disallow: /dh_
    Disallow: /category/*
    Disallow: /category/
    Disallow: /login/
    Disallow: /20*
    Disallow: /iframes/
    Disallow: /i/
    Disallow: /f/
    Disallow: /t/
    Disallow: /c/
    Disallow: /j/
    Disallow: /*.php$
    Disallow: /*?*
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /*.gz$
    Disallow: /*.wmv$
    Disallow: /*.cgi$
    Disallow: /*.xhtml$
    Disallow: /*?*
    Disallow: /*?
    
    # alexa archiver
    User-agent: ia_archiver
    Disallow: /
    
    # disable duggmirror by Digg
    User-agent: duggmirror
    Disallow: /
    
    # allow google image bot to search all images
    User-agent: Googlebot-Image
    Disallow: /wp-includes/
    Allow: /wp-content/uploads/
    
    Sitemap: http://mydomain.com/sitemap.xml.gz
     
  2. mikie46

    mikie46 Jr. VIP Jr. VIP

    Joined:
    Aug 6, 2008
    Messages:
    1,454
    Likes Received:
    1,103
    Nobody uses it huh? Interesting.
     
  3. cnick79

    cnick79 Jr. VIP Jr. VIP

    Joined:
    Jun 10, 2010
    Messages:
    805
    Likes Received:
    411
    Location:
    Wandering
    Code:
    User-agent: *
    Disallow: *
     
  4. SeRf*X

    SeRf*X Registered Member

    Joined:
    Apr 5, 2011
    Messages:
    59
    Likes Received:
    12