Best setting for robot.txt?

Discussion in 'Black Hat SEO' started by Xpired, Apr 20, 2012.

  1. Xpired

    Xpired Junior Member

    Joined:
    Jul 19, 2010
    Messages:
    196
    Likes Received:
    50
    Occupation:
    Free
    Location:
    BHW
    Like Title say...What is The Best setting for robot.txt?for wordpress,adsense..
     
  2. seoitaly

    seoitaly Newbie

    Joined:
    Apr 3, 2012
    Messages:
    13
    Likes Received:
    0
    Hi... your question it is a little bit too general.
    What is your goal?

    For example... in some cases I completely disallow the image folder because for some website I do not care at all about indexing images.... In some other cases, images bring me a lot of qualified traffic.

    Normaly with wordpress I disallow everything is not necessary from a SEO point of view like for examples: /wp-admin/ /wp-includes/ /wp-content/plugins/ /wp-content/cache/ /wp-content/themes/

    If you have on your webspace directories like stats or cgi-bin, disallow them too.I
    If you need to delete all urls with question markes (for example) from search engine indexes try to disallow also /*?

    I hope my answer was helpfull to you.