1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Robots.txt error

Discussion in 'Black Hat SEO' started by apollo_69, Jan 26, 2012.

  1. apollo_69

    apollo_69 Registered Member

    Joined:
    Jan 13, 2011
    Messages:
    95
    Likes Received:
    12
    Hi, I recently revamped one of my sites, and used Backup Buddy to copy from the test domain to the live one. I forgot to change the Privacy Settings from private, and a few days later I noticed the Title and Meta Description were missing in the SERPs completely.

    I went in and changed the Wordpress privacy settings back to public, but a few days later and that still didn't work. Then I did a bit of research and found out that WP creates a virtual robots.txt file, which you can check the contents by going to www.domain.com/robots.txt

    When I did this I saw that the file was still blocking Google's bots, so there was no way for them to find the title and meta description. So I created a physical robots.txt file and uploaded it to the root, checked www.domain.com/robots.txt and this time everything looks good.

    However a couple days later and still nothing showing in the SERPs, I'm worried if this continues much longer it will damage my ranking.....any ideas how to fix this please!?

    [​IMG]
     
  2. apollo_69

    apollo_69 Registered Member

    Joined:
    Jan 13, 2011
    Messages:
    95
    Likes Received:
    12
    Hi, bumping this one for some help please... :)
     
  3. popzzz

    popzzz Supreme Member

    Joined:
    Apr 12, 2009
    Messages:
    1,337
    Likes Received:
    13,699
    Location:
    Don't touch the REP!

    If you're going to use the WP virtual you CANNOT have a physical file or it won't work properly.
    From the sound of what you describe you did NOT turn off the virtual file in WP.

    Turn off the virtual file in WP and then add the following line to the bottom of your robots.txt file
    to your sitemap: (if not there already)
    Code:
    Sitemap: http://www.yourblog.com/sitemap.xml.gz
    (use the location of your sitemap, of course):
    Enjoy ..... :cool2:



     
  4. apollo_69

    apollo_69 Registered Member

    Joined:
    Jan 13, 2011
    Messages:
    95
    Likes Received:
    12
    Thanks Popz, the problem was the virtual robots.txt generated by Wordpress wasn't letting Google in, even after I changed the privacy settings....must be a weird bug when using Backup Buddy, something to watch out for in future....

    So I created a physical robots.txt, and in my GWT account I resubmitted the sitemap and used the Fetch as Googlebot feature on the homepage, to retrieve and resubmit to Google for indexing, that seems to have corrected the problem! Finally :)