1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Will This robots.txt File Get My Site Deindexed in Google?

Discussion in 'Black Hat SEO' started by GoTop, Jan 7, 2011.

  1. GoTop

    GoTop Junior Member

    Joined:
    Jan 31, 2009
    Messages:
    172
    Likes Received:
    16
    Hi guys, my site disappear in google suddenly three days ago.

    and I post a thread here: My One Year Old Site Disappear in Google Today, Why it Happen?

    today i login my Google Webmaster Tool, and check my robots.txt.

    this remind me three day ago, i maybe change my robots.txt setting to below:
    Code:
    User-Agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/plugins/
    Disallow: /wp-content/themes/
    I use the generate robots.txt funtion in Google Webmaster Tool, and this robots.txt is generate.
    Code:
    User-Agent: *
    Allow: /
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/plugins/
    Disallow: /wp-content/themes/
    the only difference between my robots.txt and GWT generate robots.txt is Google's example contain "Allow: /" and mine dosen't.

    is this robots.txt the reason my site deindexed in google?

    look forward for your help.
     
    Last edited: Jan 7, 2011
  2. Dumper

    Dumper Supreme Member

    Joined:
    Mar 20, 2009
    Messages:
    1,416
    Likes Received:
    499
    Location:
    Perdido Key
    The * is what matters, you don't need to put Allow: / because * means allow all except for what you Disallow:. Hopefully this was clear.
     
    • Thanks Thanks x 2
  3. GoTop

    GoTop Junior Member

    Joined:
    Jan 31, 2009
    Messages:
    172
    Likes Received:
    16
    Thanks for your explanation, but the only "*" in robots.txt is in "User-Agent: *".

    I think it mean the below rule apply for ALL User-Agent.

    and there is no other "*", could you explain more about this? Thanks.
     
  4. Gorgias

    Gorgias Newbie

    Joined:
    Sep 29, 2009
    Messages:
    5
    Likes Received:
    1
    Just have the site crawler tool generate you a new robots.txt file. In the optional section (forgive me since I don't have the screen in front of me), just click the dropdown and allow all.

    Click generate the text file. Upload the robots.txt file that the crawler tool created to your root directory and have the crawler "test" the new robots file. This should do the trick for ya.
     
    • Thanks Thanks x 1
  5. GoTop

    GoTop Junior Member

    Joined:
    Jan 31, 2009
    Messages:
    172
    Likes Received:
    16
    thanks, already done that.

    In my Google Webmaster Tools, it show that google have download my new robots.txt 18 hours ago, but my site still can't show in google yet...