1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Best setting for robot.txt?

Discussion in 'Black Hat SEO' started by Xpired, Apr 20, 2012.

  1. Xpired

    Xpired Junior Member

    Joined:
    Jul 19, 2010
    Messages:
    196
    Likes Received:
    50
    Occupation:
    Free
    Location:
    BHW
    Like Title say...What is The Best setting for robot.txt?for wordpress,adsense..
     
  2. seoitaly

    seoitaly Newbie

    Joined:
    Apr 3, 2012
    Messages:
    13
    Likes Received:
    0
    Hi... your question it is a little bit too general.
    What is your goal?

    For example... in some cases I completely disallow the image folder because for some website I do not care at all about indexing images.... In some other cases, images bring me a lot of qualified traffic.

    Normaly with wordpress I disallow everything is not necessary from a SEO point of view like for examples: /wp-admin/ /wp-includes/ /wp-content/plugins/ /wp-content/cache/ /wp-content/themes/

    If you have on your webspace directories like stats or cgi-bin, disallow them too.I
    If you need to delete all urls with question markes (for example) from search engine indexes try to disallow also /*?

    I hope my answer was helpfull to you.