Robots.txt error

apollo_69

Registered Member
Joined
Jan 13, 2011
Messages
95
Reaction score
12
Hi, I recently revamped one of my sites, and used Backup Buddy to copy from the test domain to the live one. I forgot to change the Privacy Settings from private, and a few days later I noticed the Title and Meta Description were missing in the SERPs completely.

I went in and changed the Wordpress privacy settings back to public, but a few days later and that still didn't work. Then I did a bit of research and found out that WP creates a virtual robots.txt file, which you can check the contents by going to www.domain.com/robots.txt

When I did this I saw that the file was still blocking Google's bots, so there was no way for them to find the title and meta description. So I created a physical robots.txt file and uploaded it to the root, checked www.domain.com/robots.txt and this time everything looks good.

However a couple days later and still nothing showing in the SERPs, I'm worried if this continues much longer it will damage my ranking.....any ideas how to fix this please!?

m2Col
 
Hi, bumping this one for some help please... :)
 

If you're going to use the WP virtual you CANNOT have a physical file or it won't work properly.
From the sound of what you describe you did NOT turn off the virtual file in WP.

Turn off the virtual file in WP and then add the following line to the bottom of your robots.txt file
to your sitemap: (if not there already)
Code:
Sitemap: http://www.yourblog.com/sitemap.xml.gz
(use the location of your sitemap, of course):

Enjoy ..... :cool2:



 
Thanks Popz, the problem was the virtual robots.txt generated by Wordpress wasn't letting Google in, even after I changed the privacy settings....must be a weird bug when using Backup Buddy, something to watch out for in future....

So I created a physical robots.txt, and in my GWT account I resubmitted the sitemap and used the Fetch as Googlebot feature on the homepage, to retrieve and resubmit to Google for indexing, that seems to have corrected the problem! Finally :)
 
Back
Top