Having some problems with my robots.txt file

thedon23

thedon23

Elite Member
Joined
Dec 21, 2009
Messages
1,759
Reaction score
1,271
Last night I noticed that everything on my site was being indexed (even the the wp-admin folder and all that).

I went into the root folder of my site: public_html/www.site.com

I noticed there was no robots.txt, so I created one, and added the following to it (I noticed one of the mods here had recommended this):

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Allow: /wp-content/uploads

# Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /

# digg mirror
User-agent: duggmirror
Disallow: /
Click to expand...

When I go into Google Webmaster Tools, it is still show my old robots.txt settings, which is weird because I had never even created one for the site. It shows the following:

User-agent: *
Disallow:

Sitemap: http://www.site.com/sitemap.xml.gz
Click to expand...

Then I notice, GWT is showing my robots.txt file at http://site.com/robots.txt,
whereas my robots.txt file is really at http://[B]www[/B].site.com

Anybody know how to fix this?!
 
Advertise on BHW
What is the meaning of this robots function:
Disallow: /category/*/*
and
Disallow: /*?*
Disallow: /*?

And what is the difference with:
Disallow: /category/*/*
and
Disallow: /category/
though, if you end with a "/" then it will specify that as the match.
That means this;
Disallow: /wp-includes/
will block these;
Disallow: /wp-includes/this.html
Disallow: /wp-includes/that.php
Disallow: /wp-includes/thistoo.jpg
Disallow: /wp-includes/here/here2/anythinginhere.aswell

I would be very thank full, if anybody clarify this statements.
 
To be honest, I have no idea what it means haha. I just use it because it seems like that's what most people are using.

Okay, so it looks like Google Webmaster Tools recognized my new robots.txt file. Now, like I said, all of my wp-admin folder is indexed in Google. Now that I've updated the updated the robots.txt file to not index those sites, will they soon be removed from Google's index? Or do I need to request a URL removal?
 
Bump. Anybody? I just checked, and all those useless pages are still indexed in Google.
 
BlackhatUser said:
I believe Google will remove those links soon. Give it some time :)
Click to expand...

Yeah, I did that a couple days ago. Thanks for pointing that out though. So Google sees my new robots.txt file, but all those pages are still indexed. Do I have to wait a couple of weeks or something?
 
To break it down, the root of your domain is / and then the paths are based on that. So if you have a file at .com/images/uploads/my-picture.jpg and you didn't want google to see it you'd type

Disallow: /images/uploads/my-picture.jpg

Now let's say you want to block google from indexing the entire uploads folder you'd type

Disallow: /images/uploads/*

The star means everything that uses the beginning of the path is blocked, so /images/uploads/your-picture.jpg is also blocked with the * example.

Disallow: /images/uploads/*?*

I'm pretty sure this would block any string that includes a ? followed by another string, so if that part of the extension has a ? in it it will be blocked. So it would block /images/uploads/1234?12345.html but wouldn't block /images/uploads/12345-12345.html

Disallow: /images/uploads/*/*

That would block the contents of any folder in the uploads directory but would allow indexing of the files in the uploads directory.

Pretty sure that's right, but you should google if you really want the correct answer.
 
thedon23 said:
Yeah, I did that a couple days ago. Thanks for pointing that out though. So Google sees my new robots.txt file, but all those pages are still indexed. Do I have to wait a couple of weeks or something?
Click to expand...
1. You've already taken the first step by blocking the required folders in robots txt.
2. Now, you need to request a directory (or URL) removal in Google Webmaster tools:

  • On the Webmaster Tools home page, click the site you want.
  • On the Dashboard, click Site configuration on the left-hand menu.
  • Click Crawler access, and then click Remove URL.
  • Click New removal request.
  • Type the URL of the directory you want removed from search results and then click Continue. (How to find the right URL.)
  • The URL is case-sensitive—use exactly the same characters and capitalization that the site uses.
  • If you want to remove the whole site, you can leave this blank.
  • Click Remove directory.
  • Click Submit Request.
Click to expand...
More info:
Code: 
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=1663427

The pages should drop out of the index as g00gle recrawls.
 
Last edited:
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top